Machine learning measures of student engagement correlate moderately to highly with manual ratings, but manual ratings better predict self-reported engagement¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? A machine learning measure combining head pose, gaze features, and neighbor synchrony showed a moderate to high correlation with manual ratings of student engagement. → Nückles 2021 q2 i? Manual ratings were more successful than machine learning measures in predicting students' self-reported cognitive engagement and situational interest. → Nückles 2021 (2)

Evidence¶

Nückles 2021¶

Nückles, M. (2021). Investigating Visual Perception in Teaching and Learning with Advanced Eye-Tracking Methodologies: Rewards and Challenges of an Innovative Research Paradigm. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09567-5

q2 · i? · review · r2

Gold standard study by Goldberg et al. comparing machine-learning-generated engagement measures with fine-grained second-by-second manual ratings; the algorithm was trained on surface behavioral features including gaze direction, head posture, facial expressions, and neighbor synchrony.

"the study yielded promising results showing a moderate to high correlation of the machine learning measure (combining head pose, gaze features, and neighbor synchrony) with the manual ratings of student engagement."

Nückles 2021 (2)¶

Nückles, M. (2021). Investigating Visual Perception in Teaching and Learning with Advanced Eye-Tracking Methodologies: Rewards and Challenges of an Innovative Research Paradigm. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09567-5

q2 · i? · review · r2

Same Goldberg et al. gold standard study; despite the ML measure's economy, manual ratings outperformed it for predicting self-report measures, a limit the author highlights.

"The manual ratings, however, were more successful than the machine learning measures in predicting students'self-reported cognitive engagement and situational interest."

Discussion¶

Related Claims¶

• LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks — related
• AI-based analyses of student learning may perform unevenly across student populations, limiting comparability of findings across settings — related
• Head-pose-estimation-based proctoring systems report widely varying accuracies, from 75.6% for cheating detection to 100% for yaw-angle estimation — related
• Machine learning analysis of regular video recordings can support conclusions on learners' visual perception in lectures — related
• Combining eye tracking with other process measures such as machine vision features provides useful information about student engagement — related