Combining more modalities does not always improve detection of learner mental states¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2025 · checked against its sources 2026-10-19 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? In one reviewed study, physiological plus auditory modalities better identified confusion and concentration, while fusing all three modalities better detected frustration and boredom, showing added modalities must be weighed against feasibility. → Rachelle Esterhazy 2025

Evidence¶

Rachelle Esterhazy 2025¶

Rachelle Esterhazy, Rogers Kaliisa, Daniel Sanchez, Malcolm Langford and Crina Damsa. (2025). Advancing Multimodal Collaboration Analytics: A Scoping Review. Journal of Learning Analytics 12(2). https://doi.org/10.18608/jla.2025.8625

q2 · i? · review · r2

The review reports, citing Peng and Nagao (2021), a tool using commercial sensors (iPhone, AirPods, Apple Watch) trained with supervised ML and compared against two annotators' ratings. "combining physiological and auditory modalities yield s better results for identifying st ates of confusion and concentration". No effect sizes are printed.

"The authors find that combining physiological and auditory modalities yield s better results for identifying st ates of confusion and concentration, while fusing all three modalities yields better results for detecting frustration and boredom."

Discussion¶

Related Claims¶

• Microphones are the most widely used sensor in MMCA studies — related
• Most MMCA studies combine two or three modalities rather than four — related
• Spoken text modality does not significantly differ from signaled written text modality, or from signaled written plus spoken text, for either abstract or concrete animation — related
• Certain multimodal data combinations improve predictive model performance, with audio plus eye-tracking data most effective in one K–8 study — related
• Chest-worn ECG provides the strongest cardiac-sensing performance for attention-difficulty prediction, and fusing all cardiac streams does not improve upon chest ECG alone — a narrower finding that bears on this claim
• Predictive performance of sensor configurations does not change monotonically with the number of sensing streams; preferred configuration depends on target metric and sensing cost — a narrower finding that bears on this claim