Skip to content

Multimodal analysis

Research Method · All research methods
Evidence · 12 claims (12 for) · 5 studies (4 associational, 1 design), q2–q3 · 0 of 5 report an effect size · 12 claims rest on one study

Description

Multimodal analysis is one of three methodological advances the report highlights, defined as "Integrating multiple streams of data, such as audio, video, eye gaze, sensors, and clickstream data." The introduction elaborates it as "the study of learning with data streams that include physical movement, sound, and logs from interactions with technology (as well as more traditional observations)." The report presents the Connected Worlds exhibit at the New York Hall of Science, which enables embodied learning in an immersive sensor-rich space, as a research site where scientists are pushing the frontiers of multimodal analytics.

Accounts

  • Multimodal analysis: studying learning with multiple integrated data streams: Multimodal analysis is one of three methodological advances the report highlights, defined as "Integrating multiple streams of data, such as audio, video, eye gaze, sensors, and clickstream data." The introduction elaborates it as "the study of learning with data streams that include physical movement, sound, and logs from interactions with technology (as well as more traditional observations)." The report presents the Connected Worlds exhibit at the New York Hall of Science, which enables embodied learning in an immersive sensor-rich space, as a research site where scientists are pushing the frontiers of multimodal analytics. (Roschelle et al. (2017))
  • AI-based speech processing pipeline for multi-teacher classroom audio: An automated pipeline for analysing team-teaching talk: multi-channel headset recordings are processed with a weighted fusion of spectral masking and ECAPA-TDNN speaker-embedding-based masking to reduce cross-channel interference, transcribed with WhisperX, and used to extract 88 acoustic features with openSMILE from fixed 1-second segments. Quality was verified via word error rate: "The WER decreased from158%before cross-talk mitigation to21%after processing, which is considered acceptable". Standardised features were reduced by Varimax-rotated PCA into five rotated components explaining 58% of cumulative variance. (Yuchen Liu et al. (2026))
  • EEG-fNIRS bimodal complementarity framework for attention quantification: The article's attention modeling framework rests on the complementary strengths of EEG and fNIRS. EEG offers millisecond-level temporal resolution sensitive to rapid stimulus changes but is susceptible to motion artifacts, while fNIRS suffers hemodynamic latency yet provides region-level spatial specificity and greater tolerance to movement. As the authors state, "EEG primarily contributes high-temporal-resolution neural dynamics, whereas fNIRS provides complementary region-level spatial information over the targeted cortical regions." These complementary properties motivate the bimodal input, with fused features feeding a DDQN that outputs high, medium, or low attention levels. (Shixian Zhou et al. (2026))
  • Multimodal pipeline for detecting and analyzing teacher–student group interactions in classroom video: A computational pipeline combining three modalities from high school mathematics classroom videos: OpenPose pose estimation with Euclidean-distance tracking postprocessing and group boundary boxes, openSMILE eGeMAPS acoustic/prosodic features (88 features), and WhisperX word-level transcripts aligned to detected windows. A pose-based heuristic flags teacher–group interactions using vertical dominance and proximity with 10-second smoothing and a 10-second minimum episode length. An on-premises Mistral large language model classified transcript snippets for question-asking, confusion, need for help, and math talk. (Hur et al. (2026))
  • Two-approach multi-modal learning analytics technique linking behavioral clusters to learning outcomes (Approach A backward, Approach B forward): This technique builds behavioral profiles of dyads by clustering teams on multi-modal behaviors (log, video, audio features) and then relating behavioral clusters to learning and performance outcomes. Approach A connects "the learning outcomes back to the behaviors observed during the learning process", while Approach B is "a forward approach as it helps to move from multi-modal behaviors to learning outcomes". Approach B was designed to detect multiple behavioral profiles for the same learning profile, which Approach A's one-cluster-one-profile assumption cannot do; it succeeded where Approach A found no significantly discriminating behavior between its cluster pairs. (Jauwairia Nasir et al. (2021))
  • Five emergent multimodality interaction themes characterize collaborative problem-solving across stages: The multimodal analysis surfaced five emergent interaction themes showing participants' collaborative patterns in different problem-solving stages and their different strategies. These themes capture how learners orchestrate actions on virtual objects, gestures, body movements, and speech together — for example, action on objects first followed by reasoning through language. The themes are derived from the systematic coding and episode-building of the recorded VR sessions. (Wen Huang et al. (2023))
  • Four types of collaborative learning process data for analyzing individual participation: The review synthesizes the process data used to analyze individual participation into four categories: oral conversational data (N = 28 studies, mostly co-located), textual input data (N = 18, almost all online), log data from CSCL platforms (N = 20, the most common in online settings), and nonverbal physical data (N = 14) including gestures, eye gazing, touch, and electrodermal activity. Fourteen studies employed multimodal data, most combining oral conversation data and nonverbal physical data, which the review says captured "a more comprehensive view of participation." The mapping of data types to participation forms is summarized in the study's Fig. 3. (Shihui Feng (2025))
  • Macro–meso–micro multimodal analysis framework combining classroom interaction coding with second-by-second cardiac stress measurement: The article develops a three-level analytical framework: macro-level overviews of communicative activities, meso-level visualisation of situational stress across the lesson, and micro-level second-by-second analysis of critical incidents combining discourse analysis with stress data. The authors state that "the methodological contribution of this research lies in the combination of both physiological (i.e., cardiac data) and multimodal interactional data (i.e., eye tracking, video, audio)." Stress is operationalised via the ASV derived from HRV, applying an approach from psychotherapist settings to an authentic learning environment. (Lehesvuori et al. (2026))
  • Multimodal medical-simulation coding scheme combining behavioral, SRL and SSRL codes: A composite coding scheme used to code leaders' behaviors and regulation strategies in recorded CRM simulations. As the article describes it, "Our coding scheme included codes for: 1) behaviors (posture and gestures, facial expression, and vocalics) and 2) regulation strategies, including SRL and SSRL which were then modified to fit a medically based research context." It nests subordinate codes under superordinate codes with definitions and medical-context examples, borrows established SRL and SSRL codes, and was refined with input from researchers, simulation educators and medical doctors. Three trained raters reached Cohen's Kappas of 0.71 to 0.82 before deliberation and perfect agreement after deliberation. (Moreno et al. (2024))
  • Multimodal EDA collection and SCR extraction protocol for simulation research: The study collected electrodermal activity with Empatica E4 wristbands worn on participants' non-dominant hand throughout simulations. Raw EDA was preprocessed in MATLAB's Ledalab package using a 30 Hz Butterworth low-pass filter, with artifacts removed via linear interpolation. Skin conductance responses were extracted using continuous decomposition analysis with "a threshold of 0.05 μS to identify SCRs," a 3-second delay to account for perception-to-response latency, and quantification with a 5-second moving window and 1-second step. The final physiological measure was average peaks per segment (APPS) computed per participant per simulation phase. (Moreno et al. (2025))

Claims

Key Sources

  • Roschelle, J., Martin, W., Ahn, J. & Schank, P. (Eds.). (2017). Cyberlearning Community Report: The State of Cyberlearning and the Future of Learning With Technology. Menlo Park CA: SRI International. https://digitalpromise.dspacedirect.org/items/dafbe585-c0f8-4693-bde8-a6d847a58aad
  • Hur, P., Palaguachi, C., Machaka, N., Krist, C., Dyer, E. B., D'Angelo, C., & Bosch, N. (2026). A Framework for Considering Exploration, Interpretation, and Confirmation During Data Analysis: Computationally Assisted Analysis of Teacher–Group Interactions. Journal of Educational Data Mining, 18(1). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/1030
Page metadata
Field Value
type research-method
id multimodal-analysis
aliases multimodal-classroom-video-analysis-for-teacherstudent-interaction-detection
title Multimodal analysis
description Multimodal analysis is one of three methodological advances the report highlights, defined as "Integrating multiple streams of data, such as audio, video, eye gaze, sensors, and clickstream data." The introduction elaborates it as "the study of learning with data streams that include physical moveme
status review
moved_from theory/multimodal-analysis-cyberlearning-method
generated by process:sweep-kinds on 2026-10-10
trust unverified — no one has confirmed this page's content
sources 2 — roschelle-2017, hur-2026

Frontmatter is the page's machine-readable record — see the Schema & Guide.