Since 2023, software/product papers report markedly less evaluation detail than LLM papers in review automation¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Since 2023, software/product papers average a reporting-richness score of 3.3 versus 6.3 for LLM papers and 6.0 for traditional, deep-learning, and BERT-oriented papers. → Zabaleta 2026 q2 i? 38.0% of software/product papers report no evaluation, compared with 9.3% of LLM papers and 13.6% of traditional, deep-learning, and BERT-oriented papers. → Zabaleta 2026 (2)

Evidence¶

Zabaleta 2026¶

Zabaleta, M. & Lin, B. (2026). PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews. https://arxiv.org/abs/2609.11559 doi:10.48550/arxiv.2609.11559

q2 · i? · associational · r2

Descriptive corpus comparison using the 0-15 reporting-richness index across approach groups for papers since 2023; software/product papers "average a reporting-richness score of 3.3" versus 6.3 for LLM papers.

"Since 2023, software/product papers average a reporting-richness score of 3.3, compared with 6.3 for LLM papers and 6.0 for traditional, deep-learning, and BERT-oriented papers."

Zabaleta 2026 (2)¶

Zabaleta, M. & Lin, B. (2026). PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews. https://arxiv.org/abs/2609.11559. doi:10.48550/arxiv.2609.11559

q2 · i? · associational · r2

Share of papers annotated as reporting no evaluation (the none reported category) by approach group since 2023; the contrast is descriptive, with no statistical test reported.

"38.0% of software/product papers report no evaluation, compared with 9.3% of LLM papers and 13.6% of traditional, deep-learning, and BERT-oriented papers."

Discussion¶

Related Claims¶

• Review automation has shifted toward LLM- and software-facing workflows and into review stages that can alter the evidence base — a broader claim this one bears on
• Reporting coverage increases with LLM workflow complexity, from prompt-only to engineered and retrieval/adapted/agentic methods — a narrower finding that bears on this claim
• The software-product reporting gap persists within each review stage, and reported precision and recall vary widely across approaches — related
• Reported limitation profiles differ by automation method, with prompt sensitivity especially frequent in LLM papers — related