Routinely re-estimate item parameters from consecutive adaptive test data and apply the Robust Z procedure to monitor item parameter drift¶

Strategy · All strategies
Evidence · 6 claims (6 for) · 2 studies (2 associational), q2 · 0 of 2 report an effect size · 6 claims rest on one study

Description¶

The report describes a monitoring strategy in which item parameters are periodically re-estimated from recently collected adaptive test data and compared with initial calibration values to detect drift early. It states that "One way to detect item parameter drift as early as possible is to routinely re-estimate item parameters based on adaptive test data collected consecutively" and compare them with initial values; items with significant differences may need removal or parameter updates. The Robust Z statistic, which replaces the mean with the median and the standard deviation with 0.74 × the interquartile range, is used to declare items stable or unstable.

Design Implications¶

Context¶

Requirements¶

• Requires re-estimated item parameters from recent test events to compare with original calibration values

Constraints¶

• The study applied the procedure to items with at least 1,000 test events, excluding items with less than 1,000 responses from the analysis

Target Learners¶

• Students in grades K–12 mathematics, reading, language usage, and science taking MAP Growth adaptive interim assessments

Target Learning Goals¶

• Accurate measurement of achievement and within- and between-year growth on a stable vertical (RIT) scale

Claims¶

• Robust Z procedure flagged between 4.2% and 6.9% of MAP Growth items as unstable across subjects [+W]
• 13%–15% of MAP Growth items per subject were flagged as significantly drifted by the Robust Z procedure, with no clear pattern among flagged items [+W]
• Average drift in MAP Growth item difficulty estimates is negligible across subjects, with original-new correlations above 0.99 [+W]
• Both item-level and test-level analyses indicate the MAP Growth measurement scales are remaining stable [+W]
• Students encountered substantial numbers of items with changed parameter estimates, averaging 11–38% of test length by subject [+W]
• Flagged items showed no clear pattern by original difficulty, spreading across the entire scale like unflagged items [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• He, W. (2022). MAP Growth item parameter drift study. NWEA Research Report. https://www.nwea.org/research/publication/map-growth-item-parameter-drift-study/