Lower linguistic quality of physics explanations is associated with higher odds that AI-based scoring underestimates conceptual understanding, across all eleven examined AI-based scoring approaches¶

Claim · All claims
Evidence · 1 study · associational r2 · q3 · i2 medium
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q3 i2 A one-unit decrease in expert-rated linguistic quality was associated with approximately 1.3- to 3.9-times higher odds of underestimation of conceptual understanding by AI-based scoring, with the regression coefficient negative for every approach and statistically significant (p < 0.05) in five of eleven. → Markus S. Feser & Paul L. Tschisgale 2026

Evidence¶

Markus S. Feser & Paul L. Tschisgale 2026¶

Markus S. Feser & Paul L. Tschisgale. (2026). AI-based scoring systematically underestimates conceptual understanding of linguistically weak students' explanations in physics. Physics Education, Leibniz Institute for Science and Mathematics Education, Kiel. https://orcid.org/0000-0001-8503-0951

q3 · i2 · associational · r2

Multinomial logistic regression analysis of scoring deviations (AI score minus expert score) for 116 Grade 9 physics explanations, with expert-assigned conceptual understanding as covariate. The article reports odds ratios of "1.30 ≤ 1/OR ≤ 3.85" (a medium-to-large range) and that the coefficient "was negative for every scoring approach" with p < 0.05 in five of eleven approaches.

"Overall, a one -unit decrease in linguistic quality was associated with approximately 1.3 - to 3.9 -times higher odds of underestimating conceptual understanding (1.30 ≤ 1/OR ≤ 3.85) across all scoring approaches."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• Higher linguistic quality shows no consistent association with overestimation of conceptual understanding by AI-based scoring — related
• ML-based scoring approaches more often overestimated expert-assigned scores, whereas LLM-based approaches more often underestimated them — related
• Expert-assigned conceptual understanding and linguistic quality scores are moderately correlated (r = .50) in the student explanation dataset — related
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range — related