Use item parcels (testlets), not items, as observable indicators when evaluating construct validity of CAT data¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Because CAT missingness is MNAR and item-level factor models fail to converge on CAT data, the article's results support evaluating internal structure with parcels: "by parceling items and using parcels as observable variables, the test internal structure can be recovered." Analysts should aggregate items into sub-content testlets and fit models at the parcel level, while checking for over-fitting since simpler models may fit complex-model data well.

Design Implications¶

Context¶

Requirements¶

• Testlets defined as sums of items for a given sub-content, with fit evaluated against multiple goodness-of-fit indices under both strict and relaxed criteria

Constraints¶

• Parceling over-fits models, so simpler models may spuriously fit data from more complex generating models

Target Learners¶

• K-12 students assessed with computerized adaptive tests

Target Learning Goals¶

• Establishing internal-structure validity evidence for adaptive achievement test scores

Related Strategies¶

Examples¶

-

Key Sources¶

• Shudong Wang, Hong Jiao. (2012). Examine Construct Validity of Computerized Adaptive Test in K-12 Assessments. Paper presented at the annual meeting of the National Council on Measurement in Education. https://www.nwea.org/research/publication/examine-construct-validity-of-computerized-adaptive-test-in-k-12-assessments/