MAP Growth item calibration workflow

Research Method · All research methods
Evidence · 7 claims (2 for, 4 mixed, 1 against) · 2 studies (1 associational, 1 design), q2 · 0 of 2 report an effect size · 7 claims rest on one study

Description

The existing tool, built in T-SQL within the Growth Research Database, runs an automated pipeline: data cleaning, item parameter estimation via a brute-force grid search over RIT values 100-350 that minimizes mean square fit with person abilities fixed, and a Model of Man (MoM) procedure that predicts which items need human review. Calibration samples are chosen by all grade calibration (AGC) or iterative grade range (IGR) via a chi-square goodness-of-fit test, and Pass 2 filtering removes test events with less than 10% probability of a correct response given Pass 1 estimates. The article documents this workflow as the baseline against which Pychometrik was compared.

Accounts

• The existing MAP Growth item calibration workflow: automated AGC/IGR sampling, two-pass filtering, brute-force MSF estimation, and Model of Man review triage: The existing tool, built in T-SQL within the Growth Research Database, runs an automated pipeline: data cleaning, item parameter estimation via a brute-force grid search over RIT values 100-350 that minimizes mean square fit with person abilities fixed, and a Model of Man (MoM) procedure that predicts which items need human review. Calibration samples are chosen by all grade calibration (AGC) or iterative grade range (IGR) via a chi-square goodness-of-fit test, and Pass 2 filtering removes test events with less than 10% probability of a correct response given Pass 1 estimates. The article documents this workflow as the baseline against which Pychometrik was compared. (He et al. (2021))

Claims

• Item calibration results from Pychometrik and the existing tool are mostly comparable across 1,180 AGC-calibrated MAP Growth items, with average RIT differences ranging from 0.33 (mathematics) to 0.91 (science) [+W]
• Log-likelihood comparisons indicate Pychometrik can produce more accurate item parameter estimates than the existing tool for items with RIT differences larger than 2 [-W]
• In a fixed-person calibration simulation with 60 Rasch items and 1,200 simulees, estimates from both tools were comparable to true parameters, with the existing tool underestimating difficulty by 0.03 RIT on average [+W]
• For items aligned to multiple target grades, difficulty estimates from different on-grade calibration samples differ negligibly, at most 0.1 logit on average [~W]
• At least 90% of reading and science items and about 80% of math items show difficulty differences within 0.3 logits between on-grade and all-grade calibration [~W]
• Calibration sample size differences did not affect the size of on-grade versus all-grade item difficulty estimate differences [~W]
• Item difficulty estimates from target-plus-adjacent-grade samples generally differ from original estimates about as much as on-grade estimates do, with mean differences within 0.11 logit [~W]

Related Research Methods

-

Key Sources

• He, W., Bo, E., Meyer, P., & Grandgeorge, R. (2021). A comparison of item parameter estimates in Pychometrik and the existing item calibration tool. NWEA. https://www.nwea.org/research/publication/a-comparison-of-item-parameter-estimates-in-pychometrik-and-the-existing-item-calibration-tool/

Page metadata

Field	Value
type	research-method
id	map-growth-item-calibration-workflow
title	MAP Growth item calibration workflow
description	The existing tool, built in T-SQL within the Growth Research Database, runs an automated pipeline: data cleaning, item parameter estimation via a brute-force grid search over RIT values 100-350 that minimizes mean square fit with person abilities fixed, and a Model of Man (MoM) procedure that predic
status	review
moved_from	theory/existing-map-growth-item-calibration-workflow
generated	by process:sweep-kinds on 2026-10-10
trust	unverified — no one has confirmed this page's content
sources	1 — he-2021

Frontmatter is the page's machine-readable record — see the Schema & Guide.