GPT-4 over-redacts famous names, locations, and mythological creatures that are not PII in educational forum discussions¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? GPT-4 could not reliably distinguish names of artists, scientists, or political leaders from student names, redacted country and institution names needed for course discussions, and redacted mythological creature names like Cyclops. → S. Singhal 2024

Evidence¶

S. Singhal 2024¶

S. Singhal, A. F. Zambrano, M. Pankiewicz, X. Liu, C. Porter, and R. S. Baker. (2024). De-identifying student personally identifying information with gpt-4. Proceedings of the 17th International Conference on Educational Data Mining, pages 559-565, Atlanta, Georgia, USA. https://doi.org/10.5281/zenodo.12729884

q2 · i? · design · r2

Qualitative error analysis of GPT-4 outputs: in the Poetry course, names such as "John Latouche" and "Jackson Pollock" were inappropriately redacted; in Business Trends, countries like the UK were redacted; in Mythology, the word Cyclops was redacted.

"GPT was not always able to successfully differentiate the names of artists, scientists, or political leaders from the names of students."

Discussion¶

Related Claims¶

• GPT-4 detected 45 PII words that human coders failed to redact across all nine courses — related
• GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts — a broader claim this one bears on
• GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion — related
• Low-precision PII detection disrupts the semantic integrity of educational data: Presidio and Azure AI Language false positives alter intended meaning while GPT-based models preserve it — reports the opposite
• Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall — related
• During active transfer, the application rejected learner coaching and export requests and redacted earlier project content in all 120 episodes — related
• Scientists have imposed their values of precision, objectivity, and control on language, including attempts to build machines that use and create language as humans do — related