Expert-judged sentence selection by a tuned rhetorical parser succeeded, but student reactions to key-sentence highlighting were mixed¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2 · q2
Currency · newest study 2017 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? A rhetorical parser tuned to automatically select important sentences performed very positively against a human expert's judgments on a small sample of sentences. → Passonneau 2017 q2 i? Students' perceived helpfulness of a tool highlighting key sentences in their own writing was mixed, indicating a need for explanations alongside highlighting. → Passonneau 2017 (2)

Evidence¶

Passonneau 2017¶

Passonneau, R. J., McNamara, D., Muresan, S., & Perin, D. (2017). Preface: Special Issue on Multidisciplinary Approaches to AI and Education for Reading and Writing. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-017-0158-8

q2 · i? · review · r2

The editors report, citing Knight et al., an evaluation within the participatory design of the AWA web application for law students; the parser evaluation used "a small sample of sentences (N =90)". No effect size is printed.

"On the question of whether an existing rhetorical parser from NLP can be tuned to automtically select important sentences, a small sample of sentences (N =90) judged by a human expert had very positive results."

Passonneau 2017 (2)¶

Passonneau, R. J., McNamara, D., Muresan, S., & Perin, D. (2017). Preface: Special Issue on Multidisciplinary Approaches to AI and Education for Reading and Writing. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-017-0158-8

q2 · i? · review · r2

The editors report the student-facing evaluation of AWA's highlighting feature; "the results were mixed", which the editors read as a need for explanations alongside highlighting.

"On the question of whether students would find a tool that highlights key sentences in their own writing helpful, the results were mixed, indicating a need to provide explanations to the students along with the highlighting."

Discussion¶

Related Claims¶

• LLMs and instructors agree broadly on where feedback belongs but diverge on exact sentence spans, with span overlap strongly goal-dependent and LLMs highlighting more of each essay — related
• Low-level feedback automation is appropriate only for sentence-level clarity; argumentative unclarity should trigger human review — related
• SHAP and LLM explanation methods show consistently low alignment, with mean top-10 Jaccard similarity of 0.085 — related
• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related
• Lexical adaptation of the general rhetorical parser reduced false positives in legal essay analysis without introducing parse errors — related