ASR-based item-level accuracy scores show moderate agreement with human raters in a Dutch word decoding and passage reading task¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Human and automatic accuracy measures showed moderate agreement (MCC = 0.43 word decoding, MCC = 0.55 passage reading, inter-rater MCC = 0.59). → van der Velde 2025

Evidence¶

van der Velde 2025¶

van der Velde, M., Harmsen, W., Veldkamp, B. P., Feskens, R., Keuning, J., & Swart, N. (2025). Speech Enabled Reading Fluency Assessment: a Validation Study. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00480-y

q2 · i? · design · r2

Comparison of ASR-based item-level accuracy against human annotations in the scoring inference: 333 annotated word decoding subtasks and 18 orthographically transcribed passage subtasks, scored with MCC, sensitivity and specificity. The article reports "moderate agreement".

"We found moderate agreement between human and automatic accuracy measures for the word decoding (MCC = 0.43) and passage (MCC = 0.55) reading task, and for the inter-rater agreement of the passage reading accuracy scores (MCC = 0.59)."

Discussion¶

Related Claims¶

• Reliability of the ASR-based word decoding and passage reading accuracy and speed measures is excellent, ranging from 0.96 to 1 — related
• ASR-based item-level passage reading speed scores correlate moderately to strongly with human transcriber speed scores — related
• Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• LLM-based scoring of Spanish argumentative writing achieves 60–80% per-item accuracy, slightly below human inter-rater agreement — related