The introduction reports that deep learning and NLP techniques predicted fine-grained rubric scores of student essays with success well beyond the agreement level between two human raters¶
Claim · All claims
Evidence · 1 study · reviewr1·q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Deep learning and natural language processing techniques applied to essays from a 2012 automated student assessment contest predicted human-assigned rubric scores with success well beyond the agreement level between two human raters. → Gord McCalla · Julita Vassileva 2021
Evidence¶
Gord McCalla · Julita Vassileva 2021¶
Gord McCalla · Julita Vassileva. (2021). A Festschrift to Honour Jim Greer: Guest Editors’ Introduction — A Special Issue of the International Journal of Artificial Intelligence in Education. International Journal of Artificial Intelligence in Education (2021) 31:337–356. https://doi.org/10.1007/s40593-021-00273-z
q2 · i? · review · r1
The introduction reports, describing Kumar and Boulanger's Festschrift paper on automated essay scoring using essays from a 2012 automated student assessment contest, that the techniques achieved success "well beyond the agreement level between … two human raters". No numeric agreement statistic is printed here.
"The techniques achieved success “well beyond the agreement level between … two human raters” grading the same essays."
Discussion¶
Related Claims¶
- Feature-based MLP deep learning models predict ASAP D7 rubric scores with an average QWK of 0.72 against human raters' resolved scores, exceeding the 0.60 average agreement between the two human raters — possibly the same claim (merge candidate)
- The two human raters of ASAP D7 showed an average inter-rater agreement of 0.60 QWK across rubrics, with rubric-level agreements ranging from 0.46 (Ideas–Conventions) to 0.79 (Ideas–Organization) — related
- NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
deep-learning-essay-scoring-beyond-human-rater-agreement |
title |
The introduction reports that deep learning and NLP techniques predicted fine-grained rubric scores of student essays with success well beyond the agreement level between two human raters |
description |
The introduction reports that deep learning and NLP techniques predicted fine-grained rubric scores of student essays with success well beyond the agreement level between two human raters |
status |
review |
evidence_strength |
weak |
newest_evidence |
2021 |
generated |
by process:wiki-ingest on 2026-10-28 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — gord-mccalla-julita-vassileva-2021 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.