Engagement-based metrics are more robust predictors of students' perceived feedback helpfulness than pedagogical dimensions¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? In logistic regressions predicting high perceived helpfulness, RelScore (β = 0.420, p < 0.001) and SuccScore (β = 0.187, p < 0.001) are positively and significantly associated with helpfulness, while most pedagogical dimensions show weak or inconsistent associations. → Rose Niousha 2026 q2 i? In the combined model, mistake_identification shows a significantly negative association with perceived helpfulness while providing_guidance shows a significantly positive association. → Rose Niousha 2026 (2)

Evidence¶

Rose Niousha 2026¶

Rose Niousha, Samantha Boatright Smith, Bita Akram, Peter Brusilovsky, Arto Hellas, Juho Leinonen, John DeNero, and Narges Norouzi. (2026). The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness. arXiv preprint. https://arxiv.org/abs/2605.05648 doi:10.48550/arxiv.2605.05648

q2 · i? · associational · r2

Three binary logistic regression models on student helpfulness ratings (954 ratings from 274 students under BaselineTutor; 2,915 ratings from 639 students under MisconceptionTutor), with a semester control. The quote reports "β= 0.420,p<0.001" for RelScore.

"In the engagement-only model, bothRelScore(β= 0.420,p<0.001) andSuccScore(β= 0.187,p <0.001) are positively and significantly associated with higher perceived helpfulness."

Rose Niousha 2026 (2)¶

Rose Niousha, Samantha Boatright Smith, Bita Akram, Peter Brusilovsky, Arto Hellas, Juho Leinonen, John DeNero, and Narges Norouzi. (2026). The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness. arXiv preprint. https://arxiv.org/abs/2605.05648

q2 · i? · associational · r2

Combined-model coefficients (Table 4) for pedagogical dimensions; the quote reports "β=−0.524,p= 0.038" for mistake_identification and "β= 0.349,p= 0.019" for providing_guidance.

"In the combined model,mistake_identification shows a significantly negative association (β=−0.524,p= 0.038), whereas providing_guidanceshows a significantly positive association (β= 0.349,p= 0.019)."

Discussion¶

Related Claims¶

• Pedagogically desired feedback is engaged with more (higher RelScore), but pedagogical quality shows little relation to whether actions are applied correctly (SuccScore) — related
• For revealing_answer, pedagogically undesired feedback achieves higher SuccScore in both semesters, suggesting success via copying revealed answers — related
• MisconceptionTutor achieves substantially higher feedback relevance (RelScore) than BaselineTutor on every assignment — related
• Perceived usefulness of response-level LLM feedback is most strongly associated with actionability and perceived accuracy — related
• MisconceptionTutor's advantage in feedback success (SuccScore) is mixed: substantial on earlier assignments, diminishing or reversing on later ones — related