Unifying linear and nonlinear factor analysis framework for evaluating CAT construct validity at item and parcel levels¶

Theory · All theories
Evidence · 5 claims (5 for) · 3 studies (1 causal, 1 associational, 1 theoretical), q2 · 0 of 3 report an effect size · 5 claims rest on one study

Description¶

The study's theoretical framework treats IRT models as a special case of nonlinear factor analysis within latent variable modeling, so that linear and nonlinear factor analyses can be applied in a "unifying approach that combines both linear and non-liner factor analyses." This lets the impact of missing data on test factor structure be compared at both item and item parcel/testlet levels, with Rasch, 2PL, and bifactor IRT models equivalent to item factor analysis models with equal or unequal discrimination parameters.

Design Implications¶

Context¶

Requirements¶

• Dichotomously scored items and IRT-calibrated data, with calibration via WLSMV for categorical and ML for continuous observed data in Mplus

Constraints¶

• The study did not compare the nested models (Rasch, 2PL, bifactor) using chi-square difference, AIC, or BIC statistics

Target Learners¶

• K-12 students assessed with large-scale standardized achievement tests

Target Learning Objectives¶

• Valid interpretation of achievement test scores via internal-structure construct validity evidence

Claims¶

• Cat Item Level Construct Recovery Impossible [+M]
• Parceling Testlet Level Recovers Cat Construct [+M]
• Parceling over-fits models: at the testlet level, data from more complex models fit well with simpler models [+W]
• A one-factor unidimensional model is the most reasonable fit for MAP computerized adaptive Reading and Mathematics tests across grades, terms, and ten states, though Michigan Mathematics misses the RMSEA criterion [+W]
• In an empirical comparison on a testlet-based English assessment test, the bifactor model is preferred over the second-order and unidimensional 2PL models by both AIC and BIC [+W]

Related Theories¶

• Testlet-level CFA as a method for assessing dimensionality of CAT data despite sparse item-level data

Examples¶

• Use item parcels (testlets), not items, as observable indicators when evaluating construct validity of CAT data

Key Sources¶

• Shudong Wang, Hong Jiao. (2012). Examine Construct Validity of Computerized Adaptive Test in K-12 Assessments. Paper presented at the annual meeting of the National Council on Measurement in Education. https://www.nwea.org/research/publication/examine-construct-validity-of-computerized-adaptive-test-in-k-12-assessments/