Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Claude3 and PaLM2 demonstrated substantial agreement with human graders, performing better on higher-grade assessments than on identifying failing grades. → Pecuchova 2025
Evidence¶
Pecuchova 2025¶
Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00517-2
q2 · i? · design · r2
Findings summary of the model-comparison evaluation in the software engineering grading study, as stated by the authors: the models were "excelling in higher-grade assessments but falling short in identifying failing grades".
"Models such as Claude3 and PaLM2 dem onstrated substantial agreement, excelling in higher-grade assessments but falling short in identifying failing grades."
Discussion¶
Related Claims¶
- LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
- GenAI models' grades differ significantly across models, with GPTo1 grading more leniently than Llama2, Mistral7B and Cohere — related
- Newer LLM generations do not consistently agree better with human essay graders — related
- ChatGPT's exam scores were within 10% of human grades 70% of the time in a study of AI-based grading — related
- LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — related
- Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — reports the opposite
- Even the best LLM grader reaches only moderate agreement with human teachers, below the threshold for good reliability — reports the opposite
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
claude3-palm2-substantial-agreement-failing-grades |
title |
Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades |
description |
Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades |
status |
review |
evidence_strength |
weak |
newest_evidence |
2025 |
recheck_by |
2028 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — pecuchova-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.