LoRA fine-tuning hyperparameters are not model-agnostic: increasing rank above 64 destroyed LLaMA's feedback generation while Gemma remained robust¶

Claim · All claims
Evidence · 1 study · causal r1 · q1
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q1 i? During pilot experiments, raising LoRA rank above 64 caused LLaMA to return only a numeric score and omit the JSON reasoning field entirely, while Gemma maintained both scoring accuracy and feedback quality across the same rank range. → John Maurice Gayed 2026

Evidence¶

John Maurice Gayed 2026¶

John Maurice Gayed. (2026). AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models. A PREPRINT. https://arxiv.org/abs/2606.12801 doi:10.48550/arxiv.2606.12801

q1 · i? · causal · r1

Qualitative observation from the authors' pilot experiments during fine-tuning, not a controlled benchmark. The article reports that "increasing LoRA rank above 64 caused the LLaMA model to lose its ability to produce qualitative rubric-referenced feedback."

"Specifically, during pilot experiments we observed that increasing LoRA rank above 64 caused the LLaMA model to lose its ability to produce qualitative rubric-referenced feedback: it would return a numeric score but omit the JSON reasoning field entirely, effectively “forgetting” how to follow the part of the instruction that required structured narrative output."

Discussion¶

Related Claims¶

• A LoRA-adapted open-weight Gemma-3-27B model fine-tuned on 120 essays outperforms a fine-tuned GPT-3.5 baseline on TOEFL essay scoring across all reported metrics — related
• LLaMA's LoRA adaptation compresses its prediction range to roughly 2.0–4.0, while Gemma maintains a broader distribution closer to the ground truth — related
• Under identical LoRA hyperparameters and training data, the 27B Gemma model outperforms the 70B LLaMA model on every computed essay-scoring metric — related
• Three LLMs fine-tuned on teacher-validated feedback reach reported training losses with qualitative teacher validation — related
• LoRA adaptation adds only 0.6M trainable parameters on an 86.4M frozen backbone, keeping methods lightweight relative to LLM-based scoring — related
• Adaptation through prompt injection avoids fine-tuning, so the system works with any OpenAI-compatible LLM and its logic is transparent to educators — related
• LoRA fine-tuning of Flux.1 failed to learn nuclear concepts, producing wrong-concept images on all 300 test prompts — related
• There is no universal optimal injection position for the collaborative prior; effectiveness is an architecture-sensitive design choice — related
• Increasing LoRA rank raises absolute ROUGE-L gains but produces diminishing returns in performance per million adapter parameters — related
• Model size is not the primary factor in outcome-prediction performance — related