MAB assignment overestimates effect sizes for normally distributed rewards, with greater overestimation for smaller true effects¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because a design page cites it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? With a true effect size of 0.2, prior-between simulations measured an average effect size of 0.30 at sample size m; a true effect of 0.8 was measured as 0.93. → Rafferty 2019

Evidence¶

Rafferty 2019¶

Rafferty, A. N., Ying, H., & Williams, J. J. (2019). Statistical Consequences of using Multi-armed Bandits to Conduct Adaptive Educational Experiments. Journal of Educational Data Mining, Volume 11, No 1. https://jedm.educationaldatamining.org/index.php/JEDM/article/view/357

q2 · i? · causal · r2

Simulation measurement of effect sizes from MAB-collected data for normally distributed rewards. The article reports "an average effect size of 0.30" for a true 0.2 and 0.93 for a true 0.8, attributing overestimation to measurement errors in condition means from adaptive assignment rather than filtering of nonsignificant runs.

"for small actual effect size of0.2, prior-between simula- tions found an average effect size of 0.30 with sample size m, while simulations with the large effect size of0.8 found average effect size0.93."

Discussion¶

Related Claims¶

• MAB assignment yields higher average rewards (student outcomes) than uniform random assignment during experiments — related
• MAB (Thompson sampling) assignment reduces statistical power relative to uniform random assignment, requiring roughly doubled sample sizes — related
• Over long horizons, MAB assignment places fewer total students in the less effective condition than a shorter uniform experiment followed by committing to one condition — related
• An optimistic prior distribution partially mitigates MAB power loss without substantially reducing student benefits — related
• MAB assignment increases the Type I (false positive) error rate above the nominal alpha level — related
• Type S errors (significant findings in the wrong direction) are rare under both MAB and uniform assignment — related
• Studies using a no-input control group yield significantly smaller effect sizes, protecting against overestimation — related
• In simulation, ignoring local dependence overestimates reliability, and the overestimation grows with the size of the testlet effect (17.0%-21.8% decrease under the larger testlet effect vs. 4.8%-7.4% under the small) — related
• A multi-armed bandit controller achieves scoring accuracy comparable to exhaustive grid search while reducing LLM calls by 78.4% and token consumption by 72.8% — related