Under the initial rubric with direct evaluation, machine raters systematically underestimate openness while relevance and clarity align near-perfectly¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1–r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Openness showed the largest human–machine discrepancy under the initial rubric with direct evaluation, with positive bias indicating systematic machine underestimation and 55.7% exact agreement. → Pei-Yu Tseng 2026 q2 i? Relevance and clarity showed near-perfect human–machine alignment under the initial rubric with direct evaluation. → Pei-Yu Tseng 2026 (2)

Evidence¶

Pei-Yu Tseng 2026¶

Pei-Yu Tseng, Mahir Akgun, and Peng Liu. (2026). Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation. arXiv preprint. https://arxiv.org/abs/2606.23629 doi:10.48550/arxiv.2606.23629

q2 · i? · causal · r1

Experiment 1 (initial rubric, direct evaluation) compared human instructor ratings against GPT-5.1 machine ratings on 60 sampled questions per rubric version. Openness bias was positive with agreement of "55.7%", the lowest of the four dimensions; no effect size is printed.

"openness showed the largest discrepancy: its positive bias indicates systematic underestimation by the machine, and its low agreement (55.7%) suggests it was the most difficult dimension for machine rating."

Pei-Yu Tseng 2026 (2)¶

Pei-Yu Tseng, Mahir Akgun, and Peng Liu. (2026). Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation. arXiv preprint. https://arxiv.org/abs/2606.23629

q2 · i? · causal · r2

In the same Experiment 1 comparison, relevance and clarity reached "near-perfect alignment", described as surface-tractable judgments; Table 2 reports 100.0% and 98.4% Agree%. The article prints no effect size for this contrast.

"relevance and clarity showed near-perfect alignment, as both involve Title Suppressed Due to Excessive Length 5 more surface-tractable judgments such as semantic matching and language quality"

Discussion¶

Related Claims¶

• Revising the rubric to operationalize pedagogical constructs raises human–machine agreement for openness and depth — related
• Rationale-first evaluation improves depth alignment under the initial rubric but yields only limited gains for openness — related
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range — related