OpenAI's GPT-2 Output Detector distinguished real student essays from ChatGPT-generated essays on the same subjects in a December 2022 trial¶
Claim · All claims
Evidence · 1 study · theoreticalr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Ten real student essays all scored under 1% chance of being fake while ChatGPT essays on the same subjects scored near 100%, and prompting ChatGPT to imitate an undergraduate only reduced the score to around 97%. → Debby R. E. Cotton 2024
Evidence¶
Debby R. E. Cotton 2024¶
Debby R. E. Cotton, Peter A. Cotton & J. Reuben Shipway (2024) Chatting and cheating: Ensuring academic integrity in the era of ChatGPT, Innovations in Education and Teaching International, 61:2, 228-239. https://doi.org/10.1080/14703297.2023.2190148
q2 · i? · theoretical · r2
A small trial reported in the authors' discussion: ten real student essays and ChatGPT-generated essays on the same subjects were scored by the GPT-2 Output Detector, showing "less than 1% chance that they were fake" for real work versus near-100% for generated work. No formal design or statistical test is reported.
"Ten student essays submitted by one of the authors in December 2022 all had scores indicating a less than 1% chance that they were fake, while ChatGPT generated essays on the same subjects all had scores close to 100%."
Discussion¶
Related Claims¶
- Independent evaluation found commercial AI-text detectors neither sufficiently accurate nor sufficiently reliable for high-stakes use — related
- Detection-centered academic integrity regimes are unreliable and potentially biased, making sole reliance on AI detectors unsuitable for high-stakes decisions — related
- AI-content detection tools achieve only marginally higher success than human reviewers, around 80% versus 78.4% — related
- In a documented exploratory audit, one human-authored manuscript received materially different classifications from five commercial detectors, spanning 0% human to Human Generated — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
gpt2-output-detector-distinguishes-chatgpt-essays |
title |
OpenAI's GPT-2 Output Detector distinguished real student essays from ChatGPT-generated essays on the same subjects in a December 2022 trial |
description |
OpenAI's GPT-2 Output Detector distinguished real student essays from ChatGPT-generated essays on the same subjects in a December 2022 trial |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2024 |
recheck_by |
2027 |
generated |
by process:wiki-ingest on 2026-10-16 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — debby-r-e-cotton-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.