MAP Growth grade-level item pool simulation design (on-grade, ±1-grade, and all-grade pools with on- and off-grade simulees)¶

Element · All elements
Evidence · 6 claims (6 for) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 6 claims rest on one study

Description¶

A simulation design crossing three item pool types (on-grade, ±1 grade, all-grade) with two simulee ability types (on-grade, off-grade), yielding six conditions. Simulations used the SAS sandbox tool with operational MAP Growth test specifications, Owen's Bayesian method for momentary ability estimation and MLE for final ability estimation, all on the RIT scale. "In each condition, 500 simulees were assigned to each true RIT point", with true RITs drawn from empirical 2016–2017 CCSS test event distributions in the Growth Research Database.

Design Implications¶

Context¶

Requirements¶

• Item pools aligned to CCSS for grades 2–8 in mathematics, reading, and language usage, and test event data from the Growth Research Database for the 2016–2017 school year

Constraints¶

• The number of administrations per year and longitudinal constraint used in operational test validation were not considered in this simulation study

Target Learners¶

• Simulated students in grades 2–8 with on-grade and off-grade (one grade down or up) ability levels

Target Learning Goals¶

• Measuring student achievement and growth in mathematics, reading, and language usage on the RIT scale

Claims¶

• On-grade item pools cause content balancing failures for off-grade-ability students, while all-grade pools improve balancing and language usage tests remained balanced throughout [+W]
• Testing students with off-grade ability using on-grade item pools substantially diminishes measurement precision compared with all-grade pools [+W]
• On-grade mathematics item pools yield larger SEMs and lower efficiency than all-grade pools for on-grade students, especially at the low end of the scale [+W]
• Bias and RMSE of ability estimates are larger for on-grade item pools than for ±1-grade and all-grade pools across content areas and simulee types [+W]
• Restricted item pools show higher item use rates than all-grade pools, and use rates drop sharply for off-grade simulees [+W]
• Reading SEMs exceed those in mathematics under all pool types, and only the all-grade reading pool meets the operational SEM stopping rule on average [+W]

Related Elements¶

• Enhanced CBE simulator in Assessment Integration Management (AIM)

Examples¶

• Develop grade-level item pools by adding items at the low and high ends of each grade's ability spectrum and to weak instructional areas, then validate new tests

Key Sources¶

• Li, S., & Meyer, J. P. (2019). Simulation study for evaluating MAP® Growth™ grade-level item pools. Portland, OR: NWEA. https://www.nwea.org/research/publication/simulation-study-for-evaluating-map-growth-item-pools-with-grade-level-constraints/