Inter-detector agreement between Pangram and GPTZero collapses after humanization, so a single vendor flag is not a reliable consensus signal¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Pangram and GPTZero agree moderately before humanization (Spearman ρ 0.721–0.736) but weakly afterward (ρ 0.100–0.188). → Jonathan A. Karr Jr. 2026

Evidence¶

Jonathan A. Karr Jr. 2026¶

Jonathan A. Karr Jr., Grigorii Khvatskii, Ting Hua, and Nitesh V. Chawla. (2026). Why AI Detection Fails for Academic Integrity. Proceedings of the ACM AI Leadership Summit (AILS '26). https://arxiv.org/abs/2608.11256 doi:10.48550/arxiv.2608.11256

q2 · i? · causal · r2

Agreement analysis of paired detector scores (Table 7; n = 1,224 pairs in 2013–15, n = 1,344 in 2023–25). The article reports pre-humanization Spearman ρ of 0.721 and 0.736 falling to 0.100 and 0.188 post-humanization, with Cohen's κ dropping from 0.500–0.539 to 0.200–0.221.

"Pangram and GPTZero agree moderately before humanization but weakly afterward (Table 7), so a single vendor flag is not a reliable consensus signal after evasion."

Discussion¶

Related Claims¶

• Light AI-assisted edits to human-written abstracts are flagged by commercial detectors at 64–80% (Pangram) and 38–49% (GPTZero) rates — related
• After humanization with Undetectable AI, fewer than 4% of AI-labeled rewrites remain detectable (FNR > 96%) for Pangram and GPTZero — related
• Applied to 1,163 real master's theses, Pangram flagged 529 (45.5%) as containing AI-generated content, mostly at low-to-moderate levels — related
• No detection threshold simultaneously keeps original flag rates, light-edit capture, and AI-pool false negatives low — related