On-grade item pools cause content balancing failures for off-grade-ability students, while all-grade pools improve balancing and language usage tests remained balanced throughout¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For reading tests with off-grade simulees, using the on-grade item pool caused content balancing to drop from 99.8% to 82.3%, with unbalanced areas in specific grade-pool-by-ability conditions. → Li 2019 q2 i? Language usage tests achieved perfect content balancing for simulations with both on- and off-grade simulees, with all three instructional areas balanced in all test events. → Li 2019 (2)

Evidence¶

Li 2019¶

Li, S., & Meyer, J. P. (2019). Simulation study for evaluating MAP® Growth™ grade-level item pools. Portland, OR: NWEA. https://www.nwea.org/research/publication/simulation-study-for-evaluating-map-growth-item-pools-with-grade-level-constraints/

q2 · i? · causal · r2

Content balancing analysis (Appendix E) against flag criteria for balanced, slightly unbalanced, and extremely unbalanced instructional-area item counts. Unbalanced reading conditions included the Grade 2 pool for lower Grade 1 or higher Grade 3 ability and the Grade 8 pool for lower Grade 7 ability.

"For reading, on average (assuming students with off-grade ability take the grade-level reading tests), using the on-grade item pool caused content balancing to drop from 99.8% to 82.3%."

Li 2019 (2)¶

Li, S., & Meyer, J. P. (2019). Simulation study for evaluating MAP® Growth™ grade-level item pools. Portland, OR: NWEA. https://www.nwea.org/research/publication/simulation-study-for-evaluating-map-growth-item-pools-with-grade-level-constraints/

q2 · i? · design · r2

Language usage content balancing results across all simulation conditions. By contrast, mathematics showed unbalanced results for Grades 5, 6, and 8 (e.g., the Grade 5 pool shallow in Operations and Algebraic Thinking and Geometry), and reading Grades 6–8 were weak in the RIT range 180–200 except for Vocabulary.

"Language usage tests achieved the perfect results for simulations with both on- and off-grade simulees, and all three instructional areas were balanced in all test events."

Discussion¶

Related Claims¶

• Restricted item pools show higher item use rates than all-grade pools, and use rates drop sharply for off-grade simulees — related
• Testing students with off-grade ability using on-grade item pools substantially diminishes measurement precision compared with all-grade pools — related
• Bias and RMSE of ability estimates are larger for on-grade item pools than for ±1-grade and all-grade pools across content areas and simulee types — related
• On-grade mathematics item pools yield larger SEMs and lower efficiency than all-grade pools for on-grade students, especially at the low end of the scale — related
• Reading SEMs exceed those in mathematics under all pool types, and only the all-grade reading pool meets the operational SEM stopping rule on average — related