ChatGPT in thinking mode shows the strongest agreement with human examiners on essay-style exam grading (weighted kappa 0.718)¶
Claim · All claims
Evidence · 1 study · designr2·q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? In the single-file upload setting, ChatGPT in thinking mode achieved weighted kappa 0.718 with human examiners, 50.0% exact agreement and 90.6% agreement within ±1 grade. → Brevik A 2026
Evidence¶
Brevik A 2026¶
Brevik A, Jerpseth H and Lafontan SR (2026). Large language models as grading assistants in public health education: a method-comparison study of essay-style exam assessment. Front. Educ. 11:1904452. https://doi.org/10.3389/feduc.2026.1904452
q3 · i? · design · r2
Method-comparison study grading 32 anonymized essay submissions from a master's public health course. The weighted Cohen's kappa analysis found "ChatGPT showed the strongest agreement with human examiners (weighted kappa 0.718, 95% CI 0.578–0.859)"; exact match was 50.0% and near match (±1) cumulative 90.6% (Table 2).
"In the single-file upload setting, ChatGPT showed the strongest agreement with human examiners (weighted kappa 0.718, 95% CI 0.578–0.859), followed by Kimi (weighted kappa 0.571, 95% CI 0.373–0.768)."
Discussion¶
Related Claims¶
- Kimi shows reasonable but weaker agreement with human examiners than ChatGPT (weighted kappa 0.571) — related
- LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — a broader claim this one bears on
- Gemini and Gemini Pro show the highest internal grading consistency across file upload volumes, with no clear monotonic lost-in-the-middle effect — related
- Agentic assessment systems report grading time reductions of 50–70% with 90–95% agreement with human graders — related
- ChatGPT's exam scores were within 10% of human grades 70% of the time in a study of AI-based grading — related
- LLM-based essay grading is not fully reproducible: repeated ChatGPT sessions on the same submissions show only moderate consistency (Krippendorff's alpha 0.625) — related
- LLMs use fewer extreme grades than human examiners, with grade-dependent opposing biases compressing grades toward the middle — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
chatgpt-thinking-mode-strongest-human-grading-agreement |
title |
ChatGPT in thinking mode shows the strongest agreement with human examiners on essay-style exam grading (weighted kappa 0.718) |
description |
ChatGPT in thinking mode shows the strongest agreement with human examiners on essay-style exam grading (weighted kappa 0.718) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — brevik-a-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.