The AES tool scored middle school essays more generously than teacher scorers overall and on three of four dimensions¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q3
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly

Subclaims¶

q3 i? AES overall scores (M = 8.80, SD = 3.20) were significantly higher than teacher scores (M = 7.61, SD = 2.64) on the same 160 essays, t(159) = -4.17, p < .001. → Greene Nolan 2023 q3 i? AES scored significantly higher than teachers on Support & Evidence, Organization, and Language & Style; the Claim & Focus difference showed the same trend but was not statistically significant. → Greene Nolan 2023 (2)

Evidence¶

Greene Nolan 2023¶

Greene Nolan, H. L., & Vang, M. C. (2023). Automated essay scoring in middle school writing: Understanding key predictors of students' growth and comparing artificial intelligence- and teacher-generated scores and feedback. Digital Promise. https://doi.org/10.51388/20.500.12265/187

q3 · i? · associational · r2

Paired t-test comparing 16 convened teachers' sum scores with AES sum scores on the same 160 essays; the AES mean exceeded the teacher mean by 1.19 points on the 4–16 scale.

"Compared to human teacher scorers (M = 7.61, SD = 2.64), the AES tool (M = 8.80, SD = 3.20) appeared to be more generous in scoring students overall, t(159) = -4.17, p < .001."

Greene Nolan 2023 (2)¶

Greene Nolan, H. L., & Vang, M. C. (2023). Automated essay scoring in middle school writing: Understanding key predictors of students' growth and comparing artificial intelligence- and teacher-generated scores and feedback. Digital Promise. https://doi.org/10.51388/20.500.12265/187

q3 · i? · associational · r2

Dimension-level paired tests (Table 5) found significant AES-higher differences on three dimensions; in Claim & Focus the same trend appeared but "the difference might have been statistically significant with a larger sample."

"There were statistically significant differences in AES and teacher scores for three of the four dimensions: Support & Evidence, Organization, and Language & Style, where the AES tool scored higher than teachers did (Table 5)."

Discussion¶

Related Claims¶

• In Support & Evidence and Claim & Focus, teacher and AES scores agreed at the lowest and highest rubric points but diverged in the middle, with teachers more likely to score a 2 and the AES a 3 — related
• Middle school students with the lowest baseline argumentative-writing scores showed the largest growth across essay attempts on an AES platform, while mid- and high-baseline students showed limited or no significant change
• The best AES models per FRISCO dimension reached strong agreement for Situation (QWK 0.728) and Clarity (QWK 0.763) but only fair agreement for Focus, Reason, and Inference in Indonesian physics essays
• Automated essay scoring can achieve high agreement with human raters but faces calibration constraints — a broader claim this one bears on