GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2024 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Cohen's Kappa between GPT-4 and human redactions ranged from 0.267 (Poetry) to 0.843 (Design), with lower performance in courses with longer average posts and frequent mentions of public figures or historical locations. → S. Singhal 2024
Evidence¶
S. Singhal 2024¶
S. Singhal, A. F. Zambrano, M. Pankiewicz, X. Liu, C. Porter, and R. S. Baker. (2024). De-identifying student personally identifying information with gpt-4. Proceedings of the 17th International Conference on Educational Data Mining, pages 559-565, Atlanta, Georgia, USA. https://doi.org/10.5281/zenodo.12729884
q2 · i? · design · r2
Course-level analysis of Kappa across the nine MOOCs: the article reports "The highest Kappas were observed for Design (Kappa=0.843), Accounting (Kappa=0.824)" and Poetry lowest at Kappa=0.267, linking lower performance to longer, more qualitative posts.
"GPT's performance appears to be lower in courses characterized by longer average post lengths, all of them exceeding 65 words (Poetry has a much higher average of 241 words per post)."
Discussion¶
Related Claims¶
- GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts — related
- GPT-4 over-redacts famous names, locations, and mythological creatures that are not PII in educational forum discussions — related
- GPT-4 detected 45 PII words that human coders failed to redact across all nine courses — related
- Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall — related
- MOOCs failed to deliver on their proponents' promises of transforming higher education — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
gpt4-redaction-performance-varies-by-course-content |
title |
GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion |
description |
GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2024 |
source_checked |
2026-10-20 |
recheck_by |
2027 |
generated |
by process:wiki-ingest on 2026-10-08 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — s-singhal-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.