Testlet-level CFA for assessing dimensionality of CAT data

Research Method · All research methods
Evidence · 5 claims (4 for, 1 mixed) · 2 studies (1 causal, 1 associational), q2 · 0 of 2 report an effect size · 5 claims rest on one study

Description

Because CAT administrations give different students different items, item-level factor analysis is impractical: for MAP, "typical missing rates are around 98%". The study's method treats goal scores (item clusters/testlets) as observable variables in CFA and MGCFA, since "items in each CAT test event are balanced among goal areas based on content specifications". At the testlet level the local independence assumption still holds, though it might or might not hold at the item level.

Accounts

• Testlet-level CFA as a method for assessing dimensionality of CAT data despite sparse item-level data: Because CAT administrations give different students different items, item-level factor analysis is impractical: for MAP, "typical missing rates are around 98%". The study's method treats goal scores (item clusters/testlets) as observable variables in CFA and MGCFA, since "items in each CAT test event are balanced among goal areas based on content specifications". At the testlet level the local independence assumption still holds, though it might or might not hold at the item level. (Shudong Wang et al. (2013))

Claims

• A one-factor unidimensional model is the most reasonable fit for MAP computerized adaptive Reading and Mathematics tests across grades, terms, and ten states, though Michigan Mathematics misses the RMSEA criterion [+M]
• MAP tests show metric invariance across grades for all studied tests except Michigan Mathematics tests [+M]
• Item parceling at the testlet level substantially improves model fit, allowing partial (Rasch, 2PL) or full (bifactor) recovery of CAT constructs [+W]
• CAT data cannot be fit into factor models at the item level, making item-level construct validity investigation meaningless for CAT [+W]
• Parceling over-fits models: at the testlet level, data from more complex models fit well with simpler models [~W]

Related Research Methods

• Matrix sampling with planned missingness in large-scale assessments

Key Sources

• Shudong Wang, Marty McCall, Hong Jiao, & Gregg Harris. (2013). Construct Validity and Measurement Invariance of Computerized Adaptive Testing: Application to Measures of Academic Progress (MAP) Using Confirmatory Factor Analysis. Journal of Educational and Developmental Psychology, 3(1). https://doi.org/10.5539/jedp.v3n1p88

Page metadata

Field	Value
type	research-method
id	testlet-level-cfa-for-assessing-dimensionality-of-cat-data
title	Testlet-level CFA for assessing dimensionality of CAT data
description	Because CAT administrations give different students different items, item-level factor analysis is impractical: for MAP, "typical missing rates are around 98%".
status	review
moved_from	theory/testlet-level-cfa-for-cat-data
generated	by process:sweep-kinds on 2026-10-10
trust	unverified — no one has confirmed this page's content
sources	1 — shudong-wang-2013

Frontmatter is the page's machine-readable record — see the Schema & Guide.