High agreement between two LLM screeners indicates consistency among outputs, not accuracy, because models may exhibit correlated failure patterns¶

Claim · All claims
Evidence · 1 study · review r3 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q1 i? Agreement between LLMs should be interpreted primarily as consistency among outputs, not as an independent measure of accuracy, since models may exhibit correlated failure patterns. → Eicher 2026

Evidence¶

Eicher 2026¶

Eicher, J., & da Silva, R. (2026). From Sentiment Classification to Actionable and Responsible Feedback: A Scoping Review and Evidence Map of NLP in Student Evaluation of Teaching, 2015–2026. arXiv. https://arxiv.org/abs/2609.27939 doi:10.48550/arxiv.2609.27939

q1 · i? · review · r3

Methodological argument in the review's screening section, citing validation literature on LLM-assisted screening. The review argues two LLMs should not be treated as statistically independent sources of evidence of correctness.

"Models may exhibit correlated failure patterns. High agreement between LLMs can coexist with substantially lower agreement relative to external human judgments"

Discussion¶

Related Claims¶

• Among 421 SET-NLP studies, 61.3% demonstrate output or stronger (A2+) but only 11.6% evaluate outputs with intended users or stronger (A3+), a 49.7 percentage-point drop — related
• LLM-as-judge automated evaluation can achieve human-level agreement when carefully validated — a broader claim this one bears on
• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related