Under single-pass deployment, Fs+CoT prompting yields the strongest agreement with teacher mean scores for music-analysis essay scoring (r = 0.795, ICC (2,1) = 0.657)¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Among Fs+CoT, RAG, and SC under Run1, Fs+CoT achieved the strongest agreement with teacher mean scores, with the highest Pearson r, Krippendorff's alpha, and QWK and the lowest RMSE. → Baicheng Lin 2026

Evidence¶

Baicheng Lin 2026¶

Baicheng Lin, Lingxi Jin, and Kyung-Seok Min. (2026). Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias. arXiv preprint. https://arxiv.org/abs/2608.01783 doi:10.48550/arxiv.2608.01783

q2 · i? · design · r2

Comparison of three GPT-4o-mini prompting strategies against teacher mean scores for 300 undergraduate music-analysis responses, reported in Results Table 1. Fs+CoT led with "r = 0.795, ICC (2,1) = 0.657"; RAG and SC followed at r = 0.669 and 0.545.

"Fs+CoT achieved the strongest agreement with teacher mean scores (r = 0.795, ICC (2,1) = 0.657), followed by RAG (r=0.669) and Self-Consistency (SC) (r = 0.545)."

Discussion¶

Related Claims¶

• LLM scoring performance is dimension-specific: Reasoning shows the strongest agreement and Terminology the weakest across all three strategies — related
• Median aggregation across three runs produces only minor changes in agreement and does not alter the ordering of prompting strategies — related
• Prompting strategies produce distinct directional bias profiles: Fs+CoT under-scores (bias = -1.730), RAG over-scores (bias = 1.787), and SC shows minimal directional bias (bias = -0.083) — related
• Self-Consistency is the most reproducible strategy across repeated runs yet shows weaker agreement with teacher mean scores (ICC (2,1) = 0.537) — related
• RAG prompting shows systematic leniency, over-scoring relative to teacher mean scores in all four rubric dimensions (total bias = 1.787) — related
• LLM-based essay grading is not fully reproducible: repeated ChatGPT sessions on the same submissions show only moderate consistency (Krippendorff's alpha 0.625) — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related