Multiple-choice state tests are prone to ceiling and floor effects and do not measure higher-order skills well¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2009 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2012: tool and model accuracy changes quickly

Subclaims¶

q2 i? Multiple-choice state tests may be of potentially limited value for evaluations of interventions targeting high- or low-performing students because of ceiling and floor effects. → May 2009 q2 i? Multiple-choice tests do not measure higher-order skills well, so state test scores may be inappropriate for evaluations of interventions focused on such outcomes. → May 2009

Evidence¶

May 2009¶

May, Henry, Irma Perez-Johnson, Joshua Haimson, Samina Sattar, and Phil Gleason (2009). Using State Tests in Education Experiments: A Discussion of the Issues (NCEE 2009-013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://www.mathematica.org/publications/using-state-tests-in-education-experiments-a-discussion-of-the-issues

q2 · i? · review · r2

Authors' discussion in the introduction's themes section about state tests, which consist primarily of multiple-choice items sampled broadly from state standards. The paper states this format "tends to be highest around the cut point of interest" and lower at the distribution's tail-ends.

"Multiple choice tests might be relatively more prone to ceiling and floor effects and therefore of potentially limited value for evaluations examining the effects of interventions targeting high- or low-performing students. Another concern is that multiple-choice tests do not measure higher-order skills well."

Studies bearing on this claim¶

Can selected-response items measure higher-order skills and specific domains, or mainly overall recognition-level knowledge? This table gathers the studies behind this claim and the 4 narrower claims that bear on it: 4 distinct studies (2 review, 1 associational, 1 causal), q2–q3, 0 of 4 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Reliability of state test scores is highest near cut-scores or the mean and much lower for very high- or low-performing students — related
• Test scores at the floor of a test's score scale contain real information about students' performance, so floor scores do not distort teacher value-added estimates — reports the opposite
• Some educators are concerned that students scoring at the bottom or top of a test's score distribution will negatively affect the value-added estimates of those students' teachers — related
• The lower a test's score ceiling, the more teacher value-added estimates shrink toward the average teacher, because ceiling scores contain imprecise information about achievement — a narrower finding that bears on this claim
• Increased score variability was primarily driven by spreading out at the bottom end of the achievement distribution — related
• Teachers report changing high-quality instruction to match state and interim test formats, limiting deep learning opportunities — related
• Instructional systems that do not explicitly develop evaluative expertise set up artificial but potentially removable performance ceilings — related
• The multiple-choice component of the revised ELA placement test strongly measured overall ELA performance, but few items strongly measured the three specific ELA domains — a narrower finding that bears on this claim
• State tests under NCLB measured little higher-order thinking, with only 2 percent of math items qualifying — a narrower finding that bears on this claim
• The tutor-access learning gain was concentrated in conceptual depth of written answers rather than multiple-choice scores — a narrower finding that bears on this claim
• Teaching primarily helps the student agent realize existing potential rather than raise its absolute performance ceiling — related
• Effect size measures may be perceived by teachers as telling little about the changes of real students — related