Three-criterion framework for evaluating test mode comparability

Research Method · All research methods
Evidence · 6 claims (6 for) · 1 study (1 causal), q2 · 1 of 1 report an effect size · 6 claims rest on one study

Description

The report organizes its comparability evaluation around three criteria drawn from the score-comparability literature: "there are three general categories of criteria adopted in evaluating comparability: (1) content and construct validity, (2) psychometric properties and reliability, and (3) statistical assumption/test administration condition." Content validity is checked by adherence to blueprint content specifications; the psychometric criterion covers score distributions, reliability, adaptivity, and item exposure; the administration criterion extends to students' test-taking experience via manifest variables such as test duration and length. The framework structures both the simulation

Accounts

• Three-criterion framework for evaluating test mode comparability: The report organizes its comparability evaluation around three criteria drawn from the score-comparability literature: "there are three general categories of criteria adopted in evaluating comparability: (1) content and construct validity, (2) psychometric properties and reliability, and (3) statistical assumption/test administration condition." Content validity is checked by adherence to blueprint content specifications; the psychometric criterion covers score distributions, reliability, adaptivity, and item exposure; the administration criterion extends to students' test-taking experience via manifest variables such as test duration and length. The framework structures both the simulation (Bo et al. (2020))

Claims

• Growth-score differences between CBE and COLO are small, exceeding 0.2 only in Reading Grades K–1 (favoring CBE) and Mathematics Grade 8 (favoring COLO) [+M]
• Marginal reliabilities of MAP Growth winter scores are comparable across engines and all in the 0.90s, with CBE showing slightly higher precision (lower SEM) [+M]
• The Altair (engine) main effect is not significant for winter RIT scores in either content area, but an Altair-by-sex interaction is significant in Reading only [+W]
• CBE shows better engine adaptivity than COLO, especially for extremely low or high achievement students [+W]
• Item exposure rates are comparable across engines, with most items below 10%, and the CBE Reading item-use rate is lower than COLO's [+W]
• Test lengths are comparable across engines, but CBE test durations are higher than COLO's in both terms and content areas, more prominently in Reading [+W]

Related Research Methods

-

Key Sources

• Bo, E., & Meyer, P. (2020). Comparability of MAP Growth tests administered through different technology and psychometric infrastructure: An engine evaluation study based on empirical data. Portland, OR: NWEA. https://www.nwea.org/research/publication/comparability-of-map-growth-tests-administered-through-different-technology-and-psychometric-infrastructure-an-engine-evaluation-study-based-on-empirical-data/

Page metadata

Field	Value
type	research-method
id	three-criterion-framework-for-evaluating-test-mode-comparability
title	Three-criterion framework for evaluating test mode comparability
description	The report organizes its comparability evaluation around three criteria drawn from the score-comparability literature: "there are three general categories of criteria adopted in evaluating comparability: (1) content and construct validity, (2) psychometric properties and reliability, and (3) statist
status	review
moved_from	theory/mode-comparability-three-criteria-framework
generated	by process:sweep-kinds on 2026-10-10
trust	unverified — no one has confirmed this page's content
sources	1 — bo-2020

Frontmatter is the page's machine-readable record — see the Schema & Guide.