Adaptivity and score reliability of both engines depend on item-pool depth; the CBE often performed better than COLO for extreme low or high achievers when item banks were shallow¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2021 · checked against its sources 2026-10-09 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? For tests with large item banks and broad content coverage, both engines showed acceptable adaptivity and reliable scores. → Hu 2021 q2 i? For tests with shallow item banks in certain instructional areas, both engines performed acceptably for most students except extreme low or high achievers, where the CBE often had better adaptivity and more reliable instructional area scores than COLO. → Hu 2021

Evidence¶

Hu 2021¶

Hu, A., Chien, M., & Meyer, P. (2021). Comparability of MAP Growth tests administered through different technology and psychometric infrastructure: A simulation study. NWEA. https://www.nwea.org/research/publication/comparability-of-map-growth-tests-administered-through-different-technology-and-psychometric-infrastructure-a-simulation-study/

q2 · i? · causal · r2

Simulation results comparing adaptivity (delta values) and SEM of instructional area scores across the nine tests. The report attributes non-comparable cases to "a small number of items available in a certain difficulty range or instructional area in the item pool."

"For other tests with a shallow item bank in certain instructional areas for the extreme low or high achievement students, both engines administered items with acceptable adaptivity and produced reliable scores for most students except those with extreme low or high achievement. In these extreme cases, the CBE often had better adaptivity and produced more reliable instructional area scores than COLO."

Discussion¶

Related Claims¶

• The study examines how CAT test design and item bank distribution affect content coverage and test efficiency in a CCSS-aligned reading comprehension test — related
• CBE shows better engine adaptivity than COLO, especially for extremely low or high achievement students — related
• Within a single administration, COLO administered more items and overexposed fewer items than the CBE, likely due to its randomesque exposure-control procedure — related
• The CBE and COLO engines met content specifications and recovered true scores well for the nine simulated MAP Growth tests — related
• Marginal reliabilities of MAP Growth winter scores are comparable across engines and all in the 0.90s, with CBE showing slightly higher precision (lower SEM) — related
• The adaptive algorithm's item selection produces a significantly lower SEM than fixed-form tests — related
• Integrating cognitive scores with non-cognitive learner attributes enables the system to dynamically adapt AI-generated explanations in tone, depth and instructional strategy to individual learner performance — related
• Adaptive testing yields better measurement precision than a fixed-form test of the same length — related
• Growth-score differences between CBE and COLO are small, exceeding 0.2 only in Reading Grades K–1 (favoring CBE) and Mathematics Grade 8 (favoring COLO) — related
• Across administrations, the CBE may re-administer an item to the same student before the longitudinal exposure window elapses when the item bank cannot fulfill test requirements, which COLO does not allow — related
• Differences in test-taking engagement across content categories were primarily due to differences in the reading load of items — related
• CAT adjusts test questions as the testing goes along — related