On professional OpenStax textbook summaries, matched summaries scored significantly higher than unmatched ones, with a larger effect for Content (d = 1.56) than Wording (d = 0.938)¶

Claim · All claims
Evidence · 1 study · design r2 · q2 · i3 large
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Differences between matched and unmatched summaries were statistically significant (p <.001) with large effect sizes in both Content and Wording. → Morris 2025 q2 i3 Content showed a greater effect size (d = 1.56) than Wording (d = 0.938). → Morris 2025

Evidence¶

Morris 2025¶

Morris, W., Crossley, S., Holmes, L., Ou, C., Dascalu, M., & McNamara, D. (2025). Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models. International Journal of Artificial Intelligence in Education, 35, 1022–1043. https://doi.org/10.1007/s40593-024-00395-0

q2 · i3 · design · r2

Post-hoc concurrent-validity test on 94 professional section summaries from the OpenStax Macroeconomics 2e textbook, scored matched versus randomized-unmatched to their sections by the best-performing Longformer. Matched summaries "scored higher than average in wording and content"; the printed effect sizes were d = 1.56 for Content and d = 0.938 for Wording.

"The differences between matched and unmatched content were statistically significant ( p <.001) in both domains with large effect sizes, but Content reported a greater effect size (d = 1.56) than Wording (d = 0.938)."

Discussion¶

Related Claims¶

• On new crowd-sourced Prolific summaries, the Content model generalized well (r = 0.70) but the Wording model predicted human scores poorly (r = 0.29) — related
• A principal component analysis distills the seven-criterion summary rubric into two components, Content and Wording, explaining approximately 73% of shared variance — related