In simulations, resolution metrics become strongly positively related to knowledge once guessing is possible, even when true resolution is constant¶
Claim · All claims
Evidence · 1 study · causalr2·q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Monte Carlo simulation showed gamma and da estimates change dramatically with knowledge level when guessing occurs, despite constant underlying resolution. → Vuorre 2021
Evidence¶
Vuorre 2021¶
Vuorre, M., & Metcalfe, J. (2021). Measures of relative metacognitive accuracy are confounded with task performance in tasks that permit guessing. Metacognition and Learning. https://doi.org/10.1007/s11409-020-09257-1
q2 · i? · causal · r2
Monte Carlo simulation (20,000 items per hypothetical participant, 200 replications per parameter combination) varying knowledge K, guessing probability G, and resolution R. Results shown in Fig. 3 indicate that with guessing, resolution metrics are downward biased and the bias grows as knowledge decreases; other metrics (d', phi, Pearson's r, AUCs) were similarly impacted.
"In the presence of guessing, even when the underlying resolution parameter is constant, the estimated values change dramatically depending on the level of knowledge (and hence task performance)."
Discussion¶
Related Claims¶
- Guessing biases calibration measures toward underconfidence, strongest for the lowest performers — related
- Mathematics performance is positively correlated with metacognitive resolution in multiple-choice testing across six experiments — related
- Observed performance-resolution slopes in the mathematics data were not greater than those produced by guessing in simulation — related
- Resolution metrics are smaller in 2AFC than in free-response testing despite presumably equal knowledge — related
- Performance-resolution correlation is significant in 2AFC but much smaller and mostly non-significant in free-response testing — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
simulation-guessing-biases-resolution-metrics |
title |
In simulations, resolution metrics become strongly positively related to knowledge once guessing is possible, even when true resolution is constant |
description |
In simulations, resolution metrics become strongly positively related to knowledge once guessing is possible, even when true resolution is constant |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2021 |
recheck_by |
2024 |
generated |
by process:wiki-ingest on 2026-10-18 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — vuorre-2021 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.