In Support & Evidence and Claim & Focus, teacher and AES scores agreed at the lowest and highest rubric points but diverged in the middle, with teachers more likely to score a 2 and the AES a 3¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2023 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly

Subclaims¶

q2 i? In Support & Evidence and Claim & Focus, teachers were more likely to give a 2 while the AES was more likely to give a 3, with agreement at the lowest and highest scores; Organization and Language & Style showed a different pattern. → Greene Nolan 2023 q2 i? In Organization and Language & Style, teachers skewed toward 1s and 2s while most AES scores were 2s and 3s (described as a normal distribution for Organization); for Organization, teachers attributed the pattern to holistic human reading versus line-by-line AES feedback. → Greene Nolan 2023 (2)

Evidence¶

Greene Nolan 2023¶

Greene Nolan, H. L., & Vang, M. C. (2023). Automated essay scoring in middle school writing: Understanding key predictors of students' growth and comparing artificial intelligence- and teacher-generated scores and feedback. Digital Promise. https://doi.org/10.51388/20.500.12265/187

q2 · i? · associational · r2

Domain-level score distributions (Figures 2–5) from the 160-essay convening scoring show tail agreement and middle-score divergence in Support & Evidence and Claim & Focus (not in Organization or Language & Style); teachers described erring toward indicating a student is still on their way.

"Just like in the domain of Support & Evidence, teachers were more likely to give a 2 as a score, whereas the AES appeared more likely to give a 3 as a score"

Greene Nolan 2023 (2)¶

Greene Nolan, H. L., & Vang, M. C. (2023). Automated essay scoring in middle school writing: Understanding key predictors of students' growth and comparing artificial intelligence- and teacher-generated scores and feedback. Digital Promise. https://doi.org/10.51388/20.500.12265/187

q2 · i? · associational · r2

Teachers' interpretation of the Organization distribution (Figure 4): AES reads and provides feedback line by line, so sentence-level edits can produce incoherent essays that teachers penalize but the tool rewards.

"Teachers' tendency to consider essays as 1's or 2's in Organization is likely due to the ability of a human scorer to see and grasp essays on a much more holistic level than AES systems can currently do."

Discussion¶

Related Claims¶

• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — the general claim this finding bears on, with the studies behind it

• The AES tool scored middle school essays more generously than teacher scorers overall and on three of four dimensions — related

• LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions
• Teacher ratings of language and reading skills correlate significantly with PSP-1 total scores, whereas grade point average does not
• Middle school students with the lowest baseline argumentative-writing scores showed the largest growth across essay attempts on an AES platform, while mid- and high-baseline students showed limited or no significant change — related
• Students perceive AI scoring as accurate for grammar, mechanics, and organization but skeptical of its ability to evaluate diverse styles, complex writing, tone, and emotion — related
• The GED essay is scored holistically on a six-point scale by two independent readers — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain — related