Detector flag rates on 2023–2025 original abstracts vary sharply by domain, with non-STEM rates far above STEM¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Pangram flags 2.1% of chemistry originals versus 26.6% of theology originals in 2023–2025, with GPTZero showing the same ordering (1.1% to 15.2%). → Jonathan A. Karr Jr. 2026 q2 i? Non-STEM flag rates exceed STEM rates (p < 0.001). → Jonathan A. Karr Jr. 2026

Evidence¶

Jonathan A. Karr Jr. 2026¶

Jonathan A. Karr Jr., Grigorii Khvatskii, Ting Hua, and Nitesh V. Chawla. (2026). Why AI Detection Fails for Academic Integrity. Proceedings of the ACM AI Leadership Summit (AILS '26). https://arxiv.org/abs/2608.11256 doi:10.48550/arxiv.2608.11256

q2 · i? · causal · r2

Domain-level analysis of 2023–2025 original abstracts (Table 4) scored by Pangram and GPTZero at τ=0.50. The article reports "Pangram flags 2.1% in chemistry versus 26.6% in theology" and states non-STEM rates exceed STEM (p < 0.001).

"Flag rates on 2023 to 2025 originals vary sharply by domain at the same𝜏 (Table 4): Pangram flags 2.1% in chemistry versus 26.6% in theology; GPTZero shows the same ordering (1.1% to 15.2%)."

Discussion¶

Related Claims¶

• Elevated detector scores track long-token and Academic Word List density, with numeric and non-alphabetic token ratios correlating negatively — related
• No detection threshold simultaneously keeps original flag rates, light-edit capture, and AI-pool false negatives low — related
• A detector can achieve high aggregate accuracy while producing materially different over-detection rates across author groups — related
• Applied to 1,163 real master's theses, Pangram flagged 529 (45.5%) as containing AI-generated content, mostly at low-to-moderate levels — related