In simulation, ignoring local dependence overestimates reliability, and the overestimation grows with the size of the testlet effect (17.0%-21.8% decrease under the larger testlet effect vs. 4.8%-7.4% under the small)¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2010 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2013: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Passage-based Cronbach's alpha was lower than item-based alpha in simulated testlet data, with larger decreases under the larger testlet effect condition. → Yanmei Li 2010
Evidence¶
Yanmei Li 2010¶
Yanmei Li, Shuhong Li, and Lin Wang. (2010). Application of a General Polytomous Testlet Model to the Reading Section of a Large-Scale English Language Assessment. ETS Research Report RR-10-21. http://www.ets.org/research/contact.html doi:10.1002/j.2333-8504.2010.tb02228.x
q2 · i? · design · r2
Simulation study comparing passage-based alpha (summing scored responses within a testlet) with item-based alpha across 20 simulated data sets. Under the small testlet effect condition the "proportion of decrease in reliability estimates" ranged from 4.8% to 7.4%; the report concludes reliability was overestimated when local dependence was present.
"Large decreases in reliability estimates were observed for Condition 2, and the proportion of decrease ranged from 17.0% to 21.8%."
Discussion¶
Related Claims¶
- In operational reading-test data, ignoring local dependence inflates reliability estimates, with passage-based alpha 2.3%-4.9% lower than item-based alpha across six test forms — related
- The general polytomous testlet model recovers true item parameters well in simulation, with average correlations of 0.9680, 0.8608, and 0.9982 for discrimination and difficulty parameters — related
- Ignoring testlet effects by fitting a unidimensional 2PL model produced mild shrinkage of item intercept and loading estimates toward zero relative to the bifactor model — related
- A low alpha for an error-prone covariate can attenuate, inflate, or artificially produce a spurious effect (phantom effect) in the coefficient of a perfectly reliable predictor — related
- In operational reading-test data, item parameter estimates from a standard 2PL/GPCM that ignores local dependence closely match those from the testlet model (average correlations 0.9893 and 0.9978) — related
- MAB assignment overestimates effect sizes for normally distributed rewards, with greater overestimation for smaller true effects — related
- Conventional reliability determination risks overestimating coding reliability in small group chat analysis
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ignored-local-dependence-overestimates-reliability-simulation |
title |
In simulation, ignoring local dependence overestimates reliability, and the overestimation grows with the size of the testlet effect (17.0%-21.8% decrease under the larger testlet effect vs. 4.8%-7.4% under the small) |
description |
In simulation, ignoring local dependence overestimates reliability, and the overestimation grows with the size of the testlet effect (17.0%-21.8% decrease under the larger testlet effect vs. |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2010 |
recheck_by |
2013 |
generated |
by process:wiki-ingest on 2026-09-27 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — yanmei-li-2010 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.