Hierarchical resampling simulation framework for evaluating metacognitive measures

Research Method · All research methods
Evidence · 2 claims (2 for) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description

A simulation methodology in which real confidence-judgment datasets are resampled hierarchically (participants with replacement, then trials within participants) to create realistic simulated experiments without committing to a mathematical model of confidence. The article states that "The 46 datasets from the confidence database were used to simulate 5000 experiments," with a continuous predictor sampled from a Gaussian distribution for each simulated participant. It enables false positive rate estimation for any measure computable from binary stimulus, binary response, and confidence data.

Accounts

• Hierarchical resampling simulation framework based on the confidence database for evaluating metacognitive measures: A simulation methodology in which real confidence-judgment datasets are resampled hierarchically (participants with replacement, then trials within participants) to create realistic simulated experiments without committing to a mathematical model of confidence. The article states that "The 46 datasets from the confidence database were used to simulate 5000 experiments," with a continuous predictor sampled from a Gaussian distribution for each simulated participant. It enables false positive rate estimation for any measure computable from binary stimulus, binary response, and confidence data. (Manuel Rausch (2023))

Claims

• Summary-statistic measures of metacognitive accuracy (gamma, type 2 ROC, meta-d′, meta-da) maintain a false positive rate consistent with the nominal 5% alpha level in group comparisons [+M]
• T-testing raw meta-d′/d′ or meta-da/da yields a false positive rate below 5%; log transformation or the Mann–Whitney U-test restores an adequate rate [+W]

Related Research Methods

-

Key Sources

• Manuel Rausch, Michael Zehetleitner. (2023). Evaluating false positive rates of standard and hierarchical measures of metacognitive accuracy. Metacognition and Learning. https://doi.org/10.1007/s11409-023-09353-y

Page metadata

Field	Value
type	research-method
id	hierarchical-resampling-simulation-framework-for-evaluating-metacognitive-measures
title	Hierarchical resampling simulation framework for evaluating metacognitive measures
description	A simulation methodology in which real confidence-judgment datasets are resampled hierarchically (participants with replacement, then trials within participants) to create realistic simulated experiments without committing to a mathematical model of confidence.
status	review
generated	by process:settle-candidates on 2026-10-19
trust	unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.