LLM-based essay grading is not fully reproducible: repeated ChatGPT sessions on the same submissions show only moderate consistency (Krippendorff's alpha 0.625)¶
Claim · All claims
Evidence · 1 study · designr2·q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? ChatGPT grading the same submissions on five different days yielded Krippendorff's alpha 0.625, indicating moderate session-to-session variation. → Brevik A 2026
Evidence¶
Brevik A 2026¶
Brevik A, Jerpseth H and Lafontan SR (2026). Large language models as grading assistants in public health education: a method-comparison study of essay-style exam assessment. Front. Educ. 11:1904452. https://doi.org/10.3389/feduc.2026.1904452
q3 · i? · design · r2
Repeated single-file grading by ChatGPT on five different days (Table 3 last row: alpha 0.625, 95% CI 0.445–0.735). The authors state this "is not particularly convincing" and that variation "seems higher for the middle grades than for high and low grades" (Figure 2).
"A Krippendorf's alpha of 0.63 for ChatGPT performing the same assessment on five different days is not particularly convincing, suggesting that the average or median of multiple assessments would be a prudent step in the real-world implementation of LLM-assisted grading of non-STEM university courses"
Discussion¶
Related Claims¶
- Gemini and Gemini Pro show the highest internal grading consistency across file upload volumes, with no clear monotonic lost-in-the-middle effect — related
- Within-judge variance of the repeated LLM-judge rubric is negligible, but cross-model-family agreement remains untested — related
- Under single-pass deployment, Fs+CoT prompting yields the strongest agreement with teacher mean scores for music-analysis essay scoring (r = 0.795, ICC (2,1) = 0.657) — related
- ChatGPT in thinking mode shows the strongest agreement with human examiners on essay-style exam grading (weighted kappa 0.718) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-grading-session-variability-limited-reproducibility |
title |
LLM-based essay grading is not fully reproducible: repeated ChatGPT sessions on the same submissions show only moderate consistency (Krippendorff's alpha 0.625) |
description |
LLM-based essay grading is not fully reproducible: repeated ChatGPT sessions on the same submissions show only moderate consistency (Krippendorff's alpha 0.625) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — brevik-a-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.