Favorable assessments of LLM review workflows frequently coexist with reported unmet reliability or performance requirements¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The share of high-bar concerns rises to 67% among 57 positive-with-caveats papers and 79% among 38 mixed/negative papers. → Zabaleta 2026

Evidence¶

Zabaleta 2026¶

Zabaleta, M. & Lin, B. (2026). PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews. https://arxiv.org/abs/2609.11559 doi:10.48550/arxiv.2609.11559

q2 · i? · associational · r1

Same impression-group analysis; high-bar concerns appear across all impression categories, and among 35 descriptive or unclear papers the share is 14% per the supplementary analysis.

"The corresponding share was 67% among 57 positive-with-caveats papers and 79% among 38 mixed/negative papers (Figure 4)."

Discussion¶

Related Claims¶

• Reported limitation profiles differ by automation method, with prompt sensitivity especially frequent in LLM papers — a broader claim this one bears on
• The predominant student evaluation of LLMs (42%) is positive but conditional on the student's own verification work — related
• The most frequent limitation of AI in instructional design is content reliability and transparency (16 of 28 studies, 57.14%) — related
• Standalone LLM tutors replicate Phase II's profile: strong single-interaction intelligence without systematic control over the learning trajectory — related
• First-year students report coexisting high approval of AI-enabled courses and high self-reported psychological strain, with strain indicators endorsed by more than two-thirds of the sample — related
• The most frequently cited ethical concerns about GenAI in research were hallucinated facts, fabricated references, and misleading content — related