Free-tier LLM detection performance varies widely and differs by corpus: gemini-2.0-flash leads on real data while the pattern detector leads on synthetic data for intent detection¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On the synthetic corpus the pattern detector achieved the highest F1 macro (65.59), not significantly different from gemini-2.0-flash (60.19, p=0.1741), but on the real corpus gemini-2.0-flash achieved the highest F1 macro (80.59, p<0.001 vs pattern 62.46). → Laurent Brisson 2026

Evidence¶

Laurent Brisson 2026¶

Laurent Brisson, Maria Teresa Segarra and Gregory Smits. (2026). A didactical-driven teacher assistant for a dimensional modeling course. https://arxiv.org/abs/2607.22598 doi:10.48550/arxiv.2607.22598

q2 · i? · design · r2

Intent detection evaluation (RQ2) on the real corpus (n=196 intent occurrences). Gemini-2.0-flash significantly outperformed the pattern; mistral-small (53.78, p=0.137) and openrouter-free (66.91, p=0.473) did not differ significantly from the pattern.

"Gemini-2.0-flash achieves the highest F1 macro (80.59, p<0.001 vs pattern), while the pattern reaches 62.46."

Discussion¶

Related Claims¶

• Combining the pattern detector with an LLM fallback does not improve joint detection accuracy over the LLM alone — related
• The deterministic pattern detector achieves high pair precision (73.21%) with explicit abstention but covers only 32% of real queries — related
• At natural TM prevalence (1.6%), even the best model generates roughly 4.3 false alarms per genuine detection, making fully automated stand-alone screening not viable on its own — related