Rationale-first evaluation improves depth alignment under the initial rubric but yields only limited gains for openness¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Under the initial rubric, rationale-first evaluation improved depth agreement from 70.5% to 87.0% but openness agreement only from 55.7% to 63.0%, remaining below the 83.3% achieved by rubric revision alone. → Pei-Yu Tseng 2026

Evidence¶

Pei-Yu Tseng 2026¶

Pei-Yu Tseng, Mahir Akgun, and Peng Liu. (2026). Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation. arXiv preprint. https://arxiv.org/abs/2606.23629 doi:10.48550/arxiv.2606.23629

q2 · i? · causal · r1

Experiment 3 (initial rubric, rationale-first evaluation) inserted a rubric-grounded analysis step before level assignment. The quote reports the printed agreement percentages; no effect size is printed.

"Rationale-first evaluation improved depth agreement from 70.5% to 87.0%, but openness agreement increased only from 55.7% to 63.0%, remaining substantially below the 83.3% achieved by rubric revision alone (Experiment 2)."

Discussion¶

Related Claims¶

• Combining the revised rubric with rationale-first evaluation yields the highest human–machine alignment, with dimension-specific gains — related
• Under the initial rubric with direct evaluation, machine raters systematically underestimate openness while relevance and clarity align near-perfectly — related
• Revising the rubric to operationalize pedagogical constructs raises human–machine agreement for openness and depth — related