Cross-prompt finetuning of a pre-finetuned 8B LLM on 50 instances per prompt achieves average QWK of 0.85, comparable to professional human graders (0.87)¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? With only 50 instances per prompt, cross-prompt finetuning matched or improved performance on 18 of 21 prompts and averaged 0.85 QWK versus human agreement of 0.87. → Funayama 2025

Evidence¶

Funayama 2025¶

Funayama, H., Matsubayashi, Y., Asazuma, Y., Mizumoto, T., & Inui, K. (2025). Cross-prompt Pre-finetuning of Language Models for Short Answer Scoring. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-025-00474-w

q2 · i? · causal · r2

Figure 6 experiment comparing cross-prompt finetuning (combined data from 21 target prompts) versus per-prompt finetuning of the pre-finetuned 8B model. The article reports the cross-prompt setting's average QWK of 0.85 against human inter-grader agreement of 0.87.

"For 18 out of the 21 prompts, the model performance was either the same or improved. Even though the cross-prompt finetuned model was trained on only 50 instances per prompt, it exhibited an average QWK of 0.85, which is comparable to that of professional human graders (0.87)."

Discussion¶

Related Claims¶

• LLMs cannot effectively perform in-context learning for short answer scoring, but cross-prompt pre-finetuning substantially boosts their zero-shot performance — related
• Two-phase cross-prompt pre-finetuning with key phrases significantly improves short answer scoring accuracy when in-prompt training data is limited — related
• Cross-prompt pre-finetuning shows no significant benefit when sufficient (200-answer) in-prompt training data is available — related
• Pre-finetuning performance increases with the number of distinct prompts used, holding total pre-finetuning answers fixed — related
• The pre-finetuned model learns a scoring principle: predicted scores correlate strongly and negatively with edit distance between answers and key phrases — related