Finetuned DeBERTa-v3-large models achieved human-like agreement on nine of ten NAEP math constructed response items in the held-out test set¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The grand prize-winning DeBERTa-v3-large models achieved QWK differences from human-human agreement of less than 0.05 on nine out of ten items in the held-out NAEP test set. → Morris 2025

Evidence¶

Morris 2025¶

Morris, W., Holmes, L., Choi, J. S., & Crossley, S. (2025). Automated Scoring of Constructed Response Items in Math Assessment Using Large Language Models. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-024-00418-w

q2 · i? · design · r2

The NAEP 2023 Automated Math Scoring Challenge withheld test-set scores and independently evaluated model predictions. The article reports that the approach "achieved human-like agreement (less than QWK 0.05 difference from human–human agreement) on nine out of the ten items in a held-out test set."

"This approach achieved human-like agreement (less than QWK 0.05 difference from human–human agreement) on nine out of the ten items in a held-out test set."

Discussion¶

Related Claims¶

• Cross-validated ensemble models performed better on the test set than hold-out models on internal evaluation, suggesting cross-validation leverages data more effectively — related