Blinded review identified 68% of ChatGPT-generated abstracts as chatbot-produced while 14% of authentic abstracts were misclassified as chatbot-generated¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? In Gao et al.'s evaluation, 68% of ChatGPT-generated abstracts were correctly identified (true positives) and 14% of authentic abstracts were mistakenly categorized as chatbot-generated (false positives). → Chunpeng Zhai 2024
Evidence¶
Chunpeng Zhai 2024¶
Chunpeng Zhai, Santoso Wibowo and Lily D. Li. (2024). The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review. Smart Learning Environments. https://doi.org/10.1186/s40561-024-00316-7
q2 · i? · design · r2
Study reported in the review's Table 2: Gao et al. evaluated ChatGPT's ability to generate abstracts using an AI plagiarism detection tool and critical analysis by a blinded human reviewer. The review adds that the capacity to generate credible references "might be constrained by the presence of DOIs and the accessibility of online articles".
"Of the abstracts generated by ChatGPT, 68% were correctly identified as such (true positives), while 14% of authentic abstracts were mistakenly categorized as chatbot-generated (false positives)."
Discussion¶
Related Claims¶
- A detector can achieve high aggregate accuracy while producing materially different over-detection rates across author groups — related
- AI-content detection tools achieve only marginally higher success than human reviewers, around 80% versus 78.4% — a broader claim this one bears on
- On fully AI-generated papers produced with ChatGPT Deep Research, Pangram achieved 65% strict accuracy while Turnitin, GPTZero and Copyleaks classified essentially all papers as false negatives — related
- AI-text detectors misclassify authentic TOEFL essays by non-native English speakers at a mean false-positive rate of 61.3% across seven detectors, far more often than native-speaker essays — related
- AI detectors using perplexity disproportionately misclassify non-native English writers, and the mechanism plausibly extends to other low-perplexity writers — related
- In a documented exploratory audit, one human-authored manuscript received materially different classifications from five commercial detectors, spanning 0% human to Human Generated — related
- Independent evaluation found commercial AI-text detectors neither sufficiently accurate nor sufficiently reliable for high-stakes use — a broader claim this one bears on
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
chatgpt-abstract-detection-68-14-percent |
title |
Blinded review identified 68% of ChatGPT-generated abstracts as chatbot-produced while 14% of authentic abstracts were misclassified as chatbot-generated |
description |
Blinded review identified 68% of ChatGPT-generated abstracts as chatbot-produced while 14% of authentic abstracts were misclassified as chatbot-generated |
status |
review |
evidence_strength |
weak |
newest_evidence |
2024 |
recheck_by |
2027 |
generated |
by process:wiki-ingest on 2026-10-15 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — chunpeng-zhai-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.