Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? GPT-4's recall (0.958) exceeded Farrow et al.'s combined class-list and regular-expression approach (0.905) and Holmes et al.'s fine-tuned RoBERTa (0.84), but the best supervised machine learning approach (recall 0.970, precision 0.827) still outperformed GPT-4 (precision 0.526). → S. Singhal 2024
Evidence¶
S. Singhal 2024¶
S. Singhal, A. F. Zambrano, M. Pankiewicz, X. Liu, C. Porter, and R. S. Baker. (2024). De-identifying student personally identifying information with gpt-4. Proceedings of the 17th International Conference on Educational Data Mining, pages 559-565, Atlanta, Georgia, USA. https://doi.org/10.5281/zenodo.12729884
q2 · i? · design · r2
Comparison against prior literature (Table 3) reported in the results section: GPT-4 recall of 0.958 beat Farrow et al.'s 0.905 and Holmes et al.'s 0.84, but "the current best approach using supervised machine learning algorithms" leads on both metrics.
"the current best approach using supervised machine learning algorithms [1] still outperforms GPT-4 in this task, with a recall of 0.970 and a precision of 0.827."
Discussion¶
Related Claims¶
- GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts — related
- GPT-4 over-redacts famous names, locations, and mythological creatures that are not PII in educational forum discussions — related
- Fine-tuned GPT-4o-mini achieves the highest recall (0.9589) among tested PII detection models on the CRAPII dataset — related
- Verifier models raise PII detection precision above all other tested methods but reduce recall relative to fine-tuned GPT-4o-mini — related
- GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion — related
- During active transfer, the application rejected learner coaching and export requests and redacted earlier project content in all 120 episodes — related
- GPT-4 detected 45 PII words that human coders failed to redact across all nine courses — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
gpt4-redaction-compared-to-prior-methods |
title |
Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall |
description |
Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall |
status |
review |
evidence_strength |
mixed |
newest_evidence |
2024 |
recheck_by |
2027 |
generated |
by process:wiki-ingest on 2026-10-08 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — s-singhal-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.