Combining the revised rubric with rationale-first evaluation yields the highest human–machine alignment, with dimension-specific gains¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Combining rationale-first evaluation with the revised rubric reached 94.5% openness agreement (MAE and MSE both 0.05) and 95.0% depth agreement, with no additional gain from rationale-first evaluation for depth. → Pei-Yu Tseng 2026

Evidence¶

Pei-Yu Tseng 2026¶

Pei-Yu Tseng, Mahir Akgun, and Peng Liu. (2026). Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation. arXiv preprint. https://arxiv.org/abs/2606.23629 doi:10.48550/arxiv.2606.23629

q2 · i? · causal · r2

Experiment 4 (revised rubric, rationale-first evaluation) scored the machine-generated pools against human ratings. The quote reports the printed agreement and error values; no standardised effect size is printed.

"Openness agreement reached 94.5% (MAE and MSE both 0.05). Depth agreement remained at 95.0%, with no additional gain from rationale-first evaluation."

Discussion¶

Related Claims¶

• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — the general claim this finding bears on, with the studies behind it
• Revising the rubric to operationalize pedagogical constructs raises human–machine agreement for openness and depth — related
• Rationale-first evaluation improves depth alignment under the initial rubric but yields only limited gains for openness — related
• Grammatical Accuracy is the most stable and objective scoring dimension for LLMs, while Task Response is the most challenging — related