Wise-crowd content models for summary assessment correlate well with an educational rubric, manually and via two automated NLP methods¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2017 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? A manual wise-crowd method for analyzing summary content quality and coverage correlates well with a rubric rating community college students' summaries, and two automated NLP applications of it also correlate well with the rubric. → Passonneau 2017

Evidence¶

Passonneau 2017¶

Passonneau, R. J., McNamara, D., Muresan, S., & Perin, D. (2017). Preface: Special Issue on Multidisciplinary Approaches to AI and Education for Reading and Writing. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-017-0158-8

q2 · i? · review · r2

The editors report, citing Passonneau et al., a manual content-model method derived from summaries by proficient individuals (a wise crowd); "The wise-crowed method correlates well with a rubric" and two automated NLP methods also correlate well. No coefficients are printed.

"The wise-crowed method correlates well with a rubric designed to rate summaries written by community college students. Two automated NLP methods to apply wise-crowd content assessment also correlate well with the educational rubric."

Discussion¶

Related Claims¶

• Rubric-aligned NLP features for evidence and organization outperform proxy-feature and coherence baselines and generalize across datasets — related
• Automatically coded dialogue acts correlate with manually coded collaborative activities as expected — related
• LLMs overrate short underdeveloped essays and over-penalize minor language errors in well-developed essays — related
• On new crowd-sourced Prolific summaries, the Content model generalized well (r = 0.70) but the Wording model predicted human scores poorly (r = 0.29) — related