When test dimensionality is unclear, compare several candidate calibration models to empirically justify the chosen composite scoring model¶

Strategy · All strategies
Evidence · no claims cited

Description¶

This strategy addresses practical testing situations where no clear a priori theoretical belief about the latent structure exists. The article recommends that "finding the most adequate model by comparing several candidate models provide empirical justifications of using one particular model." In the study this meant estimating one unidimensional and three two-dimensional calibrations and comparing them with AIC and chi-square difference tests before selecting the model used to produce composite scores.

Design Implications¶

Context¶

Requirements¶

• Multiple candidate models must be estimated and compared using model-fit criteria such as AIC and chi-square difference tests.

Constraints¶

• The article notes fit statistics can disagree, as happened for the lower elementary population, so the empirical justification may be ambiguous.

Target Learners¶

• Elementary school English language learners

Target Learning Goals¶

• Accurate and valid composite proficiency scores for reporting and decision making

Affordances¶

Related Strategies¶

• Explore candidate models with limited-information estimation first, then estimate the structured model with the efficient EM algorithm
• Evaluate quantisation choices empirically on the target task and hardware rather than inheriting them from other studies

Examples¶

-

Key Sources¶

• Shu Jing Yen, Leah Walker. (2007). Multidimensional IRT models for Composite Scores. Paper Presented at the 2007 Annual Meeting of the National Council of Measurement in Education, Chicago. https://eric.ed.gov/?id=ED504361