Testing students with off-grade ability using on-grade item pools substantially diminishes measurement precision compared with all-grade pools¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? SEMs were uniformly larger for off-grade simulees than on-grade simulees across all pool types, with the largest increase for the on-grade pool (SEM range 2.98–5.8 versus 2.95–3.39 for the all-grade pool in mathematics). → Li 2019

Evidence¶

Li 2019¶

Li, S., & Meyer, J. P. (2019). Simulation study for evaluating MAP® Growth™ grade-level item pools. Portland, OR: NWEA. https://www.nwea.org/research/publication/simulation-study-for-evaluating-map-growth-item-pools-with-grade-level-constraints/

q2 · i? · design · r2

Mathematics simulation conditions with off-grade simulees (3,000 test events per grade and content area; 500 simulees at each of six true RIT points one grade down or up). The all-grade pool was "1.01 to 3.42 times more efficient" than the on-grade pool for off-grade simulees.

"SEMs were uniformly larger for off-grade simulees than they were for on-grade simulees, regardless of the type of item pool, but the increase was largest for the on-grade pool"

Discussion¶

Related Claims¶

• On-grade item pools cause content balancing failures for off-grade-ability students, while all-grade pools improve balancing and language usage tests remained balanced throughout — related
• On-grade mathematics item pools yield larger SEMs and lower efficiency than all-grade pools for on-grade students, especially at the low end of the scale — related
• Reading SEMs exceed those in mathematics under all pool types, and only the all-grade reading pool meets the operational SEM stopping rule on average — related
• Bias and RMSE of ability estimates are larger for on-grade item pools than for ±1-grade and all-grade pools across content areas and simulee types — related
• Restricted item pools show higher item use rates than all-grade pools, and use rates drop sharply for off-grade simulees — related