The two human raters of ASAP D7 showed an average inter-rater agreement of 0.60 QWK across rubrics, with rubric-level agreements ranging from 0.46 (Ideas–Conventions) to 0.79 (Ideas–Organization)¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Human-human QWK agreement on D7 averaged 0.60 per rubric, and inter-rubric agreements ranged from 0.46 to 0.79, indicating moderate-strong rubric interdependence. → Kumar 2020

Evidence¶

Kumar 2020¶

Kumar, V. S., & Boulanger, D. (2020). Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined? International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-020-00211-5

q2 · i? · design · r2

Descriptive analysis of D7's hand-graded rubric scores: Table 7 prints pairwise QWK among rubrics (Ideas–Organization 0.79 highest; Ideas–Conventions 0.46 lowest) and per-rubric averages (Ideas 0.62, Organization 0.73, Style 0.69, Conventions 0.60).

"Table 7 also indicates that the rubrics generally have moderate-strong levels of agreement, suggesting that an AES system could naively predict 4 as the rubric score for each rubric and a holistic score of 16"

Discussion¶

Related Claims¶

• Feature-based MLP deep learning models predict ASAP D7 rubric scores with an average QWK of 0.72 against human raters' resolved scores, exceeding the 0.60 average agreement between the two human raters — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — reports the opposite
• The introduction reports that deep learning and NLP techniques predicted fine-grained rubric scores of student essays with success well beyond the agreement level between two human raters — related