Guessing biases calibration measures toward underconfidence, strongest for the lowest performers¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? The authors argue that correct guessing inflates performance measures, biasing calibration toward underconfidence, with the strongest bias among lowest-performing individuals. → Vuorre 2021

Evidence¶

Vuorre 2021¶

Vuorre, M., & Metcalfe, J. (2021). Measures of relative metacognitive accuracy are confounded with task performance in tasks that permit guessing. Metacognition and Learning. https://doi.org/10.1007/s11409-020-09257-1

q2 · i? · causal · r2

Analytical argument in the simulation study's discussion section: because guesses artificially inflate performance while mean judgments track knowledge, a perfectly calibrated person will appear underconfident; the authors illustrate with a chance-level 4AFC participant showing 25% observed underconfidence. They suggest estimating knowledge via Abbott's formula as one remedy.

"Guessing will bias measures of calibration towards underconfidence, such that the lowest-performing individuals will have the strongest guessing-induced underconfidence bias."

Discussion¶

Related Claims¶

• In simulations, resolution metrics become strongly positively related to knowledge once guessing is possible, even when true resolution is constant — related
• Lowest-confidence-trial accuracy in the mathematics data was at or above the guessing rate, supporting the guessing hypothesis — related
• Observed performance-resolution slopes in the mathematics data were not greater than those produced by guessing in simulation — related
• Higher-achieving students tend to be more accurate but more underconfident in calibration than lower-achieving students — related
• Learners generally underestimate how fast they will learn word pairs, and this underconfidence is larger for low-variability lists — related
• Metacognitive monitoring bias and precision modulate when students stop studying relative to their goals — related