LLM-generated essay scores correlated moderately to strongly with expert human assessor scores¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? In automated essay scoring for oral and maxillofacial surgery, the correlation between manual and GPT assessment was strong for one question and moderate for another. → Kong Mengjie 2025

Evidence¶

Kong Mengjie 2025¶

Kong Mengjie, Fok Elise Hoi Wan, Yiu Cynthia Kar Yung. (2025). A Scoping Review of Large Language Models in Dental Education: Applications, Challenges, and Prospects. International Dental Journal. https://doi.org/10.1016/j.identj.2025.103854

q2 · i? · review · r2

The review reports, citing Quah et al, that ChatGPT-4 scored essays from 69 final-year undergraduate dental students alongside 3 human assessors, with "Correlation between manual and GPT assessment: strong for question 1 and moderate for question 2."

"Correlation between manual and GPT assessment: strong for question 1 and moderate for question 2."

Discussion¶

Related Claims¶

• Agreement between AI-generated competency scores and human ratings was mixed across studies, with some reporting moderate to strong agreement and others showing score inflation and inconsistent performance — a broader claim this one bears on
• In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — reports the opposite
• Effects of emerging teaching methods vary by student grade and subject, with no significant differences in orthodontics and oral and maxillofacial surgery — related