Transformer-based NLP models achieve automated essay scoring with correlations to human grading exceeding 0.85¶

Claim · All claims
Evidence · 1 study · review r2 · q2 · i3 large
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i3 Transformer-based models such as BERT enable automated essay scoring with correlations to human grading exceeding 0.85, reducing subjectivity and grading time. → Mahamad 2025

Evidence¶

Mahamad 2025¶

Mahamad, S.; Chin, Y.H.; Zulmuksah, N.I.N.; Haque, M.M.; Shaheen, M.; Nisar, K. (2025). Technical Review: Architecting an AI-Driven Decision Support System for Enhanced Online Learning and Assessment. Future Internet 2025, 17, 383. https://doi.org/10.3390/fi17090383

q2 · i3 · review · r2

The review reports, citing Ludwig et al., that BERT-based automated essay scoring achieved a 0.85 correlation with human grading; the reviewed study is not described in design detail beyond this. The printed correlation r = 0.85 is the reported magnitude.

"Transformer-based models, such as BERT, enable automated essay scoring with correlations to human grading exceeding 0.85, reducing subjectivity and grading time [31]."

Discussion¶

Related Claims¶

• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• Benchmarking shows NLP-based automated essay scoring achieves grading times up to 80% faster than human graders while maintaining comparable accuracy — related
• The fine-tuned BERT model generalizes to an external dataset collected by a different research team, maintaining almost perfect agreement with human coders — related
• In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — related
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — a broader claim this one bears on
• A technical review reports a 70% cut in grading time and a 15% rise in learner satisfaction from NLP-based automated grading on one MOOC platform, with no comparison design described — related
• Agentic assessment systems report grading time reductions of 50–70% with 90–95% agreement with human graders — related
• The fine-tuned BERT model's classification performance is consistent across stereotype-threatened and non-threatened students and other demographic subgroups — related
• NLP-based feedback generation for textual student answers in large courses reduced grading effort by up to 85% with high precision (reported from Bernius et al.) — related