Reading SEMs exceed those in mathematics under all pool types, and only the all-grade reading pool meets the operational SEM stopping rule on average¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly

Subclaims¶

q2 i? In reading simulations, all SEMs were larger than 3; the all-grade pool had the lowest average SEM (3.4, the operational stopping rule), followed by the ±1-grade pool (3.41) and the on-grade pool (3.59), with the largest values at extreme scale ends. → Li 2019

Evidence¶

Li 2019¶

Li, S., & Meyer, J. P. (2019). Simulation study for evaluating MAP® Growth™ grade-level item pools. Portland, OR: NWEA. https://www.nwea.org/research/publication/simulation-study-for-evaluating-map-growth-item-pools-with-grade-level-constraints/

q2 · i? · causal · r2

Reading CAT simulations (40-item tests) for on-grade simulees. The on-grade pool "had the largest average SEM with a value of 3.59" and the widest range (3.33 to 5.06); values above 3.4 typically occurred at extreme ends of the scale, particularly the low end for the on-grade pool.

"The all-grade pool was the only pool that had an average SEM of 3.4 (the operational stopping rule for SEM), and it had the smallest range of SEMs (3.34 to 3.87)."

Discussion¶

Related Claims¶

• On-grade item pools cause content balancing failures for off-grade-ability students, while all-grade pools improve balancing and language usage tests remained balanced throughout — related
• Testing students with off-grade ability using on-grade item pools substantially diminishes measurement precision compared with all-grade pools — related
• On-grade mathematics item pools yield larger SEMs and lower efficiency than all-grade pools for on-grade students, especially at the low end of the scale — related
• Bias and RMSE of ability estimates are larger for on-grade item pools than for ±1-grade and all-grade pools across content areas and simulee types — related
• Restricted item pools show higher item use rates than all-grade pools, and use rates drop sharply for off-grade simulees — related