GPT-4 detected 45 PII words that human coders failed to redact across all nine courses¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? Analysis of disagreements between human and GPT-based de-identification revealed 45 PII words missed by human coders, including student names and personal webpage links. → S. Singhal 2024

Evidence¶

S. Singhal 2024¶

S. Singhal, A. F. Zambrano, M. Pankiewicz, X. Liu, C. Porter, and R. S. Baker. (2024). De-identifying student personally identifying information with gpt-4. Proceedings of the 17th International Conference on Educational Data Mining, pages 559-565, Atlanta, Georgia, USA. https://doi.org/10.5281/zenodo.12729884

q2 · i? · design · r2

Manual inspection of the 45 remaining human-GPT discrepancies in the 3,505-post dataset found "human coders failed to redact 45 words that involve PII", such as the name Laura in a Business Trends post and an unredacted personal webpage link.

"we found that after the first iteration of human-based de-identification, human coders failed to redact 45 words that involve PII distributed across all the courses"

Discussion¶

Related Claims¶

• GPT-4 achieves high recall (average 0.958) but low precision (average 0.526) when de-identifying MOOC forum posts — related
• GPT-4 over-redacts famous names, locations, and mythological creatures that are not PII in educational forum discussions — related
• GPT-4 de-identification performance varies by course, with lower agreement in courses with longer posts and more qualitative discussion — related
• Supervised machine learning de-identification still outperforms GPT-4, though GPT-4 exceeds class-list/regular-expression and transformer approaches in recall — related