Reported limitation profiles differ by automation method, with prompt sensitivity especially frequent in LLM papers¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Among LLM papers, 54% report an unmet high bar, 40% parameter or prompt sensitivity, 32% limited validation, and 45% small or narrow data; software/product papers report the same categories less often. → Zabaleta 2026

Evidence¶

Zabaleta 2026¶

Zabaleta, M. & Lin, B. (2026). PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews. https://arxiv.org/abs/2609.11559 doi:10.48550/arxiv.2609.11559

q2 · i? · associational · r2

Supplementary limitation-prevalence analysis (Supplementary Figure S2) by approach; categories are not mutually exclusive. Earlier approaches show a different profile, with small or narrow data reported by 47% of traditional-ML and 52% of BERT papers.

"Among LLM papers, 54% report an unmet high bar, 40% report parameter or prompt sensitivity, 32% report limited validation, and 45% report small or narrow data. Software/product papers report the same categories less often: 26%, 15%, 24%, and 34%, respectively."

Discussion¶

Related Claims¶

• Review automation has shifted toward LLM- and software-facing workflows and into review stages that can alter the evidence base — related
• Favorable assessments of LLM review workflows frequently coexist with reported unmet reliability or performance requirements — a narrower finding that bears on this claim
• Phase VI AI-enabled systems face six documented challenges: explainability, hallucination risk, prompt sensitivity, computational cost, validation complexity, and limited large-scale evidence — related
• Since 2023, software/product papers report markedly less evaluation detail than LLM papers in review automation — related
• The most frequent limitation of AI in instructional design is content reliability and transparency (16 of 28 studies, 57.14%) — related