ROUGE overall performance is quite poor compared with LSA, BLEU and n-gram co-occurrence for summary assessment¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2009 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? ROUGE performed poorly relative to the other three techniques in the authors' evaluation. → Automatic Summary Assessment for Intelligent Tutoring Systems 2009

Evidence¶

Automatic Summary Assessment for Intelligent Tutoring Systems 2009¶

Automatic Summary Assessment for Intelligent Tutoring Systems. (2009). Preprint submitted to Elsevier February 26, 2009. https://doi.org/10.1016/j.compedu.2009.05.008

q1 · i? · design · r2

Authors' evaluative statement from their own comparison of LSA, BLEU, n-gram co-occurrence and ROUGE, in which "the overall performance of ROUGE is quite poor compared with the other three techniques".

"After evaluating these techniques, we found that the overall perfor- mance of ROUGE is quite poor compared with the other three techniques."

Discussion¶

Related Claims¶

• An unweighted ensemble of LSA and n-gram co-occurrence achieves 96% overall accuracy in summary content assessment, outperforming LSA, n-gram, BLEU and ROUGE on five of six tests — related
• Per-topic performance varied: NP-completeness scored highest ROUGE-1 F1 (0.1285), sorting algorithms highest pedagogical quality (0.8250), and recurrence relations lowest pedagogical quality (0.6629) — related