Both fine-tuned models show regression toward the mean in score-wise bias, and error rises monotonically above score 3.0, with high scores hardest to predict¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Both models over-predict low scores (positive bias at 1.0–2.5) and underpredict high scores (negative bias at 3.5–5.0), a pattern the authors link to class imbalance; error increases monotonically above score 3.0, with Gemma RMSE 0.671 and LLaMA 0.781 at score 5.0. → John Maurice Gayed 2026

Evidence¶

John Maurice Gayed 2026¶

John Maurice Gayed. (2026). AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models. A PREPRINT. https://arxiv.org/abs/2606.12801 doi:10.48550/arxiv.2606.12801

q2 · i? · design · r2

Score-wise error analysis (RMSE, MAE, mean signed bias per ETS score level) on the 360-essay held-out set, reported in Table 5. The article notes the pattern "is exacerbated by class imbalance, scores of 1.0 and 1.5 together account for only 4 of the 360 test essays, while the 3.0 bin alone contains 103."

"Both models exhibit the classic pattern expected in constrained regression tasks: they over-predict low scores (positive bias at 1.0–2.5) and underpredict high scores (negative bias at 3.5–5.0)."

Discussion¶

Related Claims¶

• A LoRA-adapted open-weight Gemma-3-27B model fine-tuned on 120 essays outperforms a fine-tuned GPT-3.5 baseline on TOEFL essay scoring across all reported metrics — related
• LLaMA's LoRA adaptation compresses its prediction range to roughly 2.0–4.0, while Gemma maintains a broader distribution closer to the ground truth — related
• Under identical LoRA hyperparameters and training data, the 27B Gemma model outperforms the 70B LLaMA model on every computed essay-scoring metric — related
• Counterfactual fine-tuning reduces sentiment bias in LLM-generated forum replies — related
• Sentiment scores of LLM-generated replies differ significantly from human replies under both classifiers — related
• Adaptation through prompt injection avoids fine-tuning, so the system works with any OpenAI-compatible LLM and its logic is transparent to educators — related
• Small language models used as automated judges exhibit severe leniency bias against tutoring responses — related
• Fine-tuned PLMs exhibit label compression toward mid-scale QoF scores, never predicting extreme labels 1 and 7 — related
• Fine-tuned GPT-4o achieves state-of-the-art scoring of TOEFL argumentative essays with QWK 0.84 and RMSE 0.44 against official scores — related
• LLMs overrate short underdeveloped essays and over-penalize minor language errors in well-developed essays — related
• Self-hosted open-weight LLMs offer privacy, reproducibility, and environmental advantages over commercial LLM interfaces in educational ITSs — related
• LLM feedback sentiment tracks assigned scores: low scores come with more criticism, high scores with more praise — related
• Fine-tuned PLMs outperform prompted LLMs in rubric-based teaching quality scoring accuracy, while LLMs better preserve score variability — related