Parceling over-fits models: at the testlet level, data from more complex models fit well with simpler models¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2012 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2015: tool and model accuracy changes quickly

Subclaims¶

q2 i? At the testlet level, models tend to over-fit data, such that data generated from more complex models can be fitted well with simpler models (e.g., Rasch fitting 2PL- and bifactor-generated data). → Shudong Wang 2012

Evidence¶

Shudong Wang 2012¶

Shudong Wang, Hong Jiao. (2012). Examine Construct Validity of Computerized Adaptive Test in K-12 Assessments. Paper presented at the annual meeting of the National Council on Measurement in Education. https://www.nwea.org/research/publication/examine-construct-validity-of-computerized-adaptive-test-in-k-12-assessments/

q2 · i? · causal · r2

Simulation cross-calibration results (Table 8): each of the three models' data were calibrated with the other two models; the article concludes "The study shows that parceling over-fits model."

"However, at the testlet level, models tend to over-fit data in which data from more complex models can be fitted well with the simpler model. For example, the Rasch model fits data generated from 2PL and bifactor models well."

Discussion¶

Related Claims¶

• Item parceling at the testlet level substantially improves model fit, allowing partial (Rasch, 2PL) or full (bifactor) recovery of CAT constructs — related
• In operational reading-test data, item parameter estimates from a standard 2PL/GPCM that ignores local dependence closely match those from the testlet model (average correlations 0.9893 and 0.9978) — related
• In an empirical comparison on a testlet-based English assessment test, the bifactor model is preferred over the second-order and unidimensional 2PL models by both AIC and BIC — related
• Ignoring testlet effects by fitting a unidimensional 2PL model produced mild shrinkage of item intercept and loading estimates toward zero relative to the bifactor model — related
• CAT data cannot be fit into factor models at the item level, making item-level construct validity investigation meaningless for CAT — related