GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Across nine MOOCs, GPT-4 identified almost all PII (recall consistently over 0.85 per course, average 0.958) but frequently over-redacted non-PII names and locations (precision below 0.75 in all courses, average 0.526). → S. Singhal 2024
Evidence¶
S. Singhal 2024¶
S. Singhal, A. F. Zambrano, M. Pankiewicz, X. Liu, C. Porter, and R. S. Baker. (2024). De-identifying student personally identifying information with gpt-4. Proceedings of the 17th International Conference on Educational Data Mining, pages 559-565, Atlanta, Georgia, USA. https://doi.org/10.5281/zenodo.12729884
q2 · i? · design · r2
Evaluation of GPT-4 redaction against corrected human ground truth on 3,505 forum posts from nine MOOCs, averaged over three runs. The article reports "The recall rate was consistently over 0.85 for all courses examined" and precision below 0.75 in all cases; Table 2 gives averages of 0.958 recall and 0.526 precision.
"The recall rate was consistently over 0.85 for all courses examined. However, precision was lower than 0.75 in all cases. These results show that GPT identified almost all PII."
Discussion¶
Related Claims¶
- GPT-4 detected 45 PII words that human coders failed to redact across all nine courses — related
- Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall — related
- Fine-tuned GPT-4o-mini achieves the highest recall (0.9589) among tested PII detection models on the CRAPII dataset — related
- GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion — related
- GPT-4 over-redacts famous names, locations, and mythological creatures that are not PII in educational forum discussions — a narrower finding that bears on this claim
- During active transfer, the application rejected learner coaching and export requests and redacted earlier project content in all 120 episodes — related
- MOOCs failed to deliver on their proponents' promises of transforming higher education — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
gpt4-high-recall-low-precision-pii-redaction |
title |
GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts |
description |
GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2024 |
recheck_by |
2027 |
generated |
by process:wiki-ingest on 2026-10-08 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — s-singhal-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.