Taxonomy of four calibration and scoring approaches for multi-timepoint survey measures¶

Theory · All theories
Evidence · 2 claims (1 for, 1 mixed) · 2 studies (1 causal, 1 associational), q2 · 0 of 2 report an effect size · 2 claims rest on one study

Description¶

The article organizes scoring options for repeated survey administration into four approaches ordered by complexity: (1) cross-sectional IRT calibration with a unidimensional model using a single timepoint, (2) longitudinal unidimensional calibration pooling all timepoints, (3) longitudinal multidimensional IRT (MIRT) calibration with latent variables for each timepoint, and (4) a two-tier MIRT adding specific factors for serial correlation from repeated items. The article frames the core choice: "whether to use an IRT model that calibrates based on a single timepoint versus a MIRT model that includes latent variables for scores at all timepoints." Unlike Approaches 1 and 2, the MIRT approach explicitly accounts for over-time correlations.

Design Implications¶

Context¶

Requirements¶

• Item responses from the same Likert-type items administered across multiple timepoints
• Specification of calibration sample, IRT model type, and scoring method (e.g., EAP)

Constraints¶

• The two-tier approach does not model serial correlation due to each item being administered repeatedly across timepoints in Approach 3's formulation; Approach 1 assumes items function similarly across ages

Target Learners¶

• elementary and middle school students surveyed on psychological and social-emotional constructs

Target Learning Objectives¶

• accurate estimation of student growth trajectories on latent psychological and social-emotional constructs

Claims¶

• Multidimensional IRT scores that account for latent variable covariances over time produce better recovery of growth parameters in longitudinal growth models than sum scores and other IRT approaches [+W]
• Response styles can affect estimates of growth parameters, including the slope, on social-emotional survey constructs [~W]

Related Theories¶

Examples¶

-

Key Sources¶

• Kuhfeld, Megan, and James Soland. (2020). Estimating Student Growth on Psychological and Social-emotional Constructs: A Comparison of Multiple Scoring Approaches. EdWorkingPaper No. 20-193. https://www.edworkingpapers.com/ai20-193