The proposed AES system predicts holistic essay scores through predicted rubric scores, achieving a QWK of 0.78, competitive with cutting-edge AES tools reporting 0.77–0.81¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The system's holistic score prediction via rubric scores achieved QWK 0.78, within the 0.77–0.81 range reported for cutting-edge AES tools. → Kumar 2020

Evidence¶

Kumar 2020¶

Kumar, V. S., & Boulanger, D. (2020). Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined? International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-020-00211-5

q2 · i? · design · r2

Abstract-reported headline result of the rubric-based holistic scoring pipeline evaluated on ASAP D7; the article states the 0.77–0.81 comparison figures were "computed as per the same procedure as in this article".

"Further, the AES system proposed in this article predicts holistic essay scores through its predicted rubric scores and produces a QWK of 0.78, a competitive performance according to recent literature where cutting-edge AES tools generate agreement levels between 0.77 and 0.81"

Discussion¶

Related Claims¶

• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Automated essay scoring can achieve high agreement with human raters but faces calibration constraints — related
• Feature-based MLP deep learning models predict ASAP D7 rubric scores with an average QWK of 0.72 against human raters' resolved scores, exceeding the 0.60 average agreement between the two human raters — related
• The best AES models per FRISCO dimension reached strong agreement for Situation (QWK 0.728) and Clarity (QWK 0.763) but only fair agreement for Focus, Reason, and Inference in Indonesian physics essays — related