BLEU-4 scored zero on all 179 mathematical-proof responses, which the authors interpret as a property of n-gram metrics rather than system failure¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? BLEU-4 was zero across all 179 questions because logically equivalent proofs share almost no n-grams when notation, variable names, or proof strategy differ. → Sushan Adhikari 2026

Evidence¶

Sushan Adhikari 2026¶

Sushan Adhikari. (2026). AlgoRAG: Retrieval-Augmented Generation for Theoretical Computer Science Education — A Comprehensive Evaluation Framework for Algorithm Analysis and Complexity Theory. arXiv preprint. https://arxiv.org/abs/2609.14572 doi:10.48550/arxiv.2609.14572

q2 · i? · design · r2

Evaluation of all 179 AlgoRAG responses against instructor reference answers found BLEU-4 of 0.0000 on every question. The article states this shows "mathematical proofs routinely use logically equivalent but lexically distinct formulations", so n-gram precision fails for proof evaluation.

"BLEU-4 is zero across all questions, confirming that mathematical proofs routinely use logically equivalent but lexically distinct formulations."

Discussion¶

Related Claims¶

• AlgoRAG answered all 179 TCS exam-style questions successfully with a mean response time of 38.0 seconds and a pedagogical quality score of 0.7620 — a broader claim this one bears on
• Per-topic performance varied: NP-completeness scored highest ROUGE-1 F1 (0.1285), sorting algorithms highest pedagogical quality (0.8250), and recurrence relations lowest pedagogical quality (0.6629) — related
• The design strained on later proof-heavy topics, and students asked for better artifacts rather than fewer tests — related
• Cluster membership showed a strong multivariate effect on standard text-similarity metrics (Pillai's trace = 0.85, F(12, 248) = 15.31, p < 0.001) — related