The pre-finetuned model learns a scoring principle: predicted scores correlate strongly and negatively with edit distance between answers and key phrases¶

Claim · All claims
Evidence · 1 study · causal r2 · q2 · i3 large
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 In zero-shot settings, normalized edit distance between key phrases and annotated justification cues correlated negatively with predicted scores (r = -0.79 and r = -0.83 for the two best prompts). → Funayama 2025

Evidence¶

Funayama 2025¶

Funayama, H., Matsubayashi, Y., Asazuma, Y., Mizumoto, T., & Inui, K. (2025). Cross-prompt Pre-finetuning of Language Models for Short Answer Scoring. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-025-00474-w

q2 · i3 · causal · r2

Zero-shot behavioral analysis of the pre-finetuned key-phrase model on the two best-performing prompts (QWK 0.81 and 0.79), relating normalized edit distance to justification cues and predicted scores, with correlation coefficients of r = -0.79 and r = -0.83. The article concludes the model partially grasps that answers gain higher scores when they include expressions semantically closer to the key phrases.

"The correlation coefficients for Y14_2-2_2_3-B and Y14_1-2_1_3-D were -0.79 and -0.83, respectively, indicating a strong negative correlation between the edit distance and predicted scores."

Discussion¶

Related Claims¶

• Cross-prompt finetuning of a pre-finetuned 8B LLM on 50 instances per prompt achieves average QWK of 0.85, comparable to professional human graders (0.87) — related
• LLMs cannot effectively perform in-context learning for short answer scoring, but cross-prompt pre-finetuning substantially boosts their zero-shot performance — related
• Pre-finetuning without key phrases does not improve scoring performance and slightly lowers it relative to the BERT baseline — related
• Two-phase cross-prompt pre-finetuning with key phrases significantly improves short answer scoring accuracy when in-prompt training data is limited — related