Exam questions that GPT-4 answered inconsistently or incorrectly were harder for human examinees, who scored about 63–68% on them versus about 75% on the rest¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? For Exam 1, students averaged 63.0% (SD = 10.5%) and 67.7% (SD = 12.0%) on the questions GPT-4 answered inconsistently or incorrectly, versus 75.8% (SD = 12.7%) and 75.3% (SD = 13.2%) on the remaining questions. → Joost C. F. de Winter 2024
Evidence¶
Joost C. F. de Winter 2024¶
Joost C. F. de Winter. (2024). Can ChatGPT Pass High School Exams on English Language Comprehension? International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00372-z
q2 · i? · associational · r2
Secondary correlational analysis using publicly available psychometric data for Exam 1 (CITO, 2022), comparing student performance on the 7 (GPT-4-0314) and 9 (GPT-4-0613) inconsistent or incorrectly answered questions against the remaining items. "questions that ChatGPT answered inconsistently or incorrectly appeared to pose greater challenges to human examinees".
"students achieved an average score of 63.0% ( SD = 10.5%) and 67.7% (SD = 12.0%), compared to 75.8% (SD = 12.7%) and 75.3% (SD = 13.2%) for the remaining questions (n = 36, n = 34)."
Discussion¶
Related Claims¶
- Bootstrapped self-consistency at temperature 1.0 flags potentially incorrect GPT-4 answers: about half of inconsistent responses were incorrect versus about 5% of consistent ones — related
- GPT-3.5 performed worst on open one-point exam questions, while GPT-4 improved mainly on multiple-choice items — related
- ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
- GPT-4 significantly outperforms the average Dutch student on VWO English reading comprehension, scoring 8.3 (March) and 8.1 (June) without re-prompting — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-28 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.