In operational reading-test data, ignoring local dependence inflates reliability estimates, with passage-based alpha 2.3%-4.9% lower than item-based alpha across six test forms¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2010 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2013: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Passage-based reliability was lower than item-based reliability in all six real data sets, indicating reliability was overestimated when local dependence was ignored. → Yanmei Li 2010
Evidence¶
Yanmei Li 2010¶
Yanmei Li, Shuhong Li, and Lin Wang. (2010). Application of a General Polytomous Testlet Model to the Reading Section of a Large-Scale English Language Assessment. ETS Research Report RR-10-21. http://www.ets.org/research/contact.html doi:10.1002/j.2333-8504.2010.tb02228.x
q2 · i? · design · r2
Reliability analysis of the six operational reading data sets (Data A-F), comparing item-based and passage-based Cronbach's alpha. The report also found "higher test information at middle range of ability levels were found using the 2PL/GPCM, in which local dependence was ignored" for all six data sets.
"The proportion of decrease in reliability estimates by computing the passage-based alpha ranged from 2.3% to 4.9%, suggesting that the reliability was overestimated for these data if local dependence is ignored."
Discussion¶
Related Claims¶
- In simulation, ignoring local dependence overestimates reliability, and the overestimation grows with the size of the testlet effect (17.0%-21.8% decrease under the larger testlet effect vs. 4.8%-7.4% under the small) — related
- In operational reading-test data, item parameter estimates from a standard 2PL/GPCM that ignores local dependence closely match those from the testlet model (average correlations 0.9893 and 0.9978) — related
- A multidimensional IRT model with simple structure fits testlet-based reading data better than both the general polytomous testlet model and the standard 2PL/GPCM — related
- PPMC-N detects local dependence in dichotomous IRT models comparably to the Bayesian PPMC approach — related
- Ignoring testlet effects by fitting a unidimensional 2PL model produced mild shrinkage of item intercept and loading estimates toward zero relative to the bifactor model — related
- Thorndike Scale Alpha appears to test comprehension of the questions as much as comprehension of the passage read — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
real-data-reliability-overestimated-2-3-to-4-9-percent |
title |
In operational reading-test data, ignoring local dependence inflates reliability estimates, with passage-based alpha 2.3%-4.9% lower than item-based alpha across six test forms |
description |
In operational reading-test data, ignoring local dependence inflates reliability estimates, with passage-based alpha 2.3%-4.9% lower than item-based alpha across six test forms |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2010 |
recheck_by |
2013 |
generated |
by process:wiki-ingest on 2026-09-27 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — yanmei-li-2010 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.