Computerized adaptive test (CAT) designs as a test-design dimension affecting coverage and efficiency¶

Theory · All theories
Evidence · 3 claims (3 for) · 3 studies (3 causal), q1–q2 · 0 of 3 report an effect size · 3 claims rest on one study

Description¶

Computerized adaptive test (CAT) designs are treated in the article as a test-design dimension whose choices, together with item bank distribution, shape two outcomes of a reading comprehension assessment: content coverage and efficiency. The article frames CAT design as something that can be varied experimentally, investigating "the effects of CAT test design and bank distribution on the content coverage and the efficiency of the tests." The available text does not elaborate the specific CAT algorithms compared.

Design Implications¶

Context¶

Requirements¶

• An item bank whose distribution can be characterized and varied
• Outcome measures for content coverage and test efficiency

Constraints¶

Target Learners¶

• Students taking adaptive reading comprehension assessments

Target Learning Objectives¶

• Efficient measurement of reading comprehension with adequate content coverage

Claims¶

• Cat Design And Bank Distribution Affect Coverage And Efficiency [+W]
• Adaptivity and score reliability of both engines depend on item-pool depth; the CBE often performed better than COLO for extreme low or high achievers when item banks were shallow [+W]
• On-grade mathematics item pools yield larger SEMs and lower efficiency than all-grade pools for on-grade students, especially at the low end of the scale [+W]

Related Theories¶

• Testlet-level CFA as a method for assessing dimensionality of CAT data despite sparse item-level data
• A CAT-sensitive process for evaluating content alignment

Examples¶

-

Key Sources¶

• Shudong Wang, Hong Jiao. (2016). Effects of computerized adaptive test designs on content coverage and efficiency of reading comprehension test with passages aligned to Common Core State Standards. Presentation paper. https://www.nwea.org/research/publication/effects-of-computerized-adaptive-test-designs-on-content-coverage-and-efficiency-of-reading-comprehension-test-with-passages-aligned-to-common-core-state-standards/