Rubric-aligned NLP features for evidence and organization outperform proxy-feature and coherence baselines and generalize across datasets¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2017 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Features directly representing components of evidence and organization rubrics performed better than baseline models relying on proxy features or existing text-coherence methods, and generalized well across datasets. → Passonneau 2017

Evidence¶

Passonneau 2017¶

Passonneau, R. J., McNamara, D., Muresan, S., & Perin, D. (2017). Preface: Special Issue on Multidisciplinary Approaches to AI and Education for Reading and Writing. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-017-0158-8

q2 · i? · review · r2

The editors report, citing Rahimi et al., NLP applied to a rubric for text-based writing addressing use of evidence and organization of ideas; "the methods developed here perform better and generalize well across datasets". No statistics are printed.

"In comparison to baseline models that rely on proxy features, or that adopt existing methods for analysis of text coherence, e.g., Barzilay and Lapata ( 2005) and Morris and Hirst (1991), the methods developed here perform better and generalize well across datasets."

Discussion¶

Related Claims¶

• Rubric-grounded prompting alone provides limited proficiency control, aligning far better on content-oriented traits than on form-oriented traits — related
• Wise-crowd content models for summary assessment correlate well with an educational rubric, manually and via two automated NLP methods — related
• On new crowd-sourced Prolific summaries, the Content model generalized well (r = 0.70) but the Wording model predicted human scores poorly (r = 0.29) — a narrower finding that bears on this claim