Use multi-method, multi-timing measurement and subscale-level correlational analysis when evaluating metacognition¶

Strategy · All strategies
Evidence · 3 claims (3 for) · 3 studies (1 quant-synthesis, 1 associational, 1 theoretical), q2 · 0 of 3 report an effect size · 3 claims rest on one study

Description¶

The article's methodological recommendations for metacognition research are drawn from the moderators it discusses. Researchers agree "a multi-method approach using both on-line and off-line tasks is required" to understand relationships between metacognitive components. For timing, some researchers suggest employing concurrent and retrospective task-specific measures for more reliable measurement. For analysis, it has been suggested that correlational analysis be carried out by corresponding subscales instead of overall scores, since different measures may evaluate different subcomponents.

Design Implications¶

Context¶

Requirements¶

• Studies comparing metacognition measures must use pure metacognition measures that do not include other skills such as motivation, and use within-design correlational comparisons.

Constraints¶

• The article notes not all skills are easily measured concurrently; for example, evaluating one's performance must by its nature be measured retrospectively.

Target Learners¶

• researchers and practitioners evaluating metacognitive skills in students and general populations

Target Learning Goals¶

• accurate measurement of metacognitive knowledge, regulation, and their relation to behavior

Claims¶

• Self-report factors correlate weakly with metacognitive behavior while within-factor subcomponent correlations are larger, per studies reviewed for the meta-analysis [+M]
• At scale level, the questionnaire's Elaboration & evaluation scale correlated significantly with the same think-aloud scale (r=.60) while the Orientation & planning scale did not (r=.24) [+W]
• Schellings' questionnaire-thinking-aloud convergent correlation of .51 rests on a small sample and is questionable against other studies [+W]

Related Strategies¶

• Use large samples, true experimental designs, and moderate treatment durations when evaluating SRL scaffolds
• Develop online measures of students' strategy use, linking learning analytics with retrospective task reports
• Use a multi-study mixed method design combining process data (eye tracking), concurrent think-aloud, and retrospective interviews to study SRL adaptation

Examples¶

-

Key Sources¶

• Craig, K., Hale, D., Grainger, C., & Stewart, M. E. (2020). Evaluating metacognitive self-reports: systematic reviews of the value of self-report in metacognitive research. Metacognition and Learning, 15, 155-213. https://doi.org/10.1007/s11409-020-09222-y