Metacognition assessment is shifting from self-report questionnaires toward process-oriented multimodal methods, which face validity and data-alignment challenges¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q1
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2023: tool and model accuracy changes quickly

Subclaims¶

q1 i? There has been a recent surge in research using process-oriented methods such as combining concurrent think-louds, log files, eye tracking, and screen recordings, though researchers continue to grapple with validity, reliability, missing data, and temporally aligning data. → Roger Azevedo 2020

Evidence¶

Roger Azevedo 2020¶

Roger Azevedo. (2020). Reflections on the field of metacognition: issues, challenges, and opportunities. Metacognition and Learning. https://doi.org/10.1007/s11409-020-09231-x

q1 · i? · theoretical · r2

The editor's observational assessment of methodological trends over the last decade: while self-report questionnaires, interviews, and observations remain widely used, process-oriented multimodal methods are surging and transitioning from laboratory studies to classrooms, homes, workplaces, and advanced learning technologies. No quantitative trend data are printed.

"there has been a recent surge in research using process-oriented methods such as combining concurrent think-louds, log files, eye tracking, screen recordings of learner-system interactions and so forth"

Discussion¶

Related Claims¶

• Integrated multimodal SRL studies are increasing, but motivational and affective processes remain underrepresented and temporal alignment of data streams is rarely done — related
• An equal number of reviewed studies measured mind wandering unobtrusively via multimodal data streams, with 27% using machine learning — related
• Questionnaire data may capture learners' perceptions rather than the strategies they actually performed — related
• Off-line self-report instruments show low concurrent validity with on-line measures and do not predict learning outcomes very well — possibly the same claim (merge candidate)
• Distinguishing metacognitive knowledge from skills, and metacognition from self-regulated learning, is an unresolved field-wide issue — related
• Mind wandering can both prompt SRL and function as a regulatory strategy, and multimodal machine-learning sensing offers scalable assessment of attentive states — related
• Think-aloud reports are incomplete and sometimes nonveridical, particularly for metacognitive processes — related
• Research on teachers' SRL knowledge is methodologically dominated by cross-sectional quantitative designs with non-standardized measures — related
• Machine-learning analysis of multimodal classroom indicators can predict student productivity, and prior knowledge becomes relevant later in the regulation process — related
• Motivational and affective dimensions remain underrepresented in multimodal SRL research despite theoretical consensus on their importance — related
• Multimodal SRL research shows a historical shift from unimodal toward integrated analytical designs, but motivational and affective dimensions remain underrepresented — related
• Standard descriptive statistics dominate SRL multimodal analysis even in integrated approaches — related
• Eye tracking can be used to study student engagement, teacher expertise, and teacher-student interaction in naturalistic classroom contexts — related
• Multimodal data collection is increasingly common but analyses rarely align data streams temporally — possibly the same claim (merge candidate)