Suitability and feasibility framework for deciding whether to use state tests in education experiments¶
Research Method · All research methods
Evidence · 5 claims (5 for) · 1 study (1 theoretical),q2· 0 of 1 report an effect size · 5 claims rest on one study
Description¶
The paper organizes the decision of whether to use state assessment data in an RCT around two concepts from measurement theory. Suitability concerns whether the assessment yields accurate, useful information about intervention effects, and is further divided into validity (the degree to which the assessment measures the outcomes targeted by the intervention) and reliability (the degree to which scores are free of random measurement error). Feasibility concerns the practical task of gaining access to the necessary state test data. The authors note the two concepts are interdependent: an unreliable score is too noisy to be valid, and a reliable measure that misses the outcome of interest is no
Accounts¶
- Suitability and feasibility framework for deciding whether to use state tests in education experiments: The paper organizes the decision of whether to use state assessment data in an RCT around two concepts from measurement theory. Suitability concerns whether the assessment yields accurate, useful information about intervention effects, and is further divided into validity (the degree to which the assessment measures the outcomes targeted by the intervention) and reliability (the degree to which scores are free of random measurement error). Feasibility concerns the practical task of gaining access to the necessary state test data. The authors note the two concepts are interdependent: an unreliable score is too noisy to be valid, and a reliable measure that misses the outcome of interest is no (May et al. (2009))
Claims¶
- Reliability of state test scores is highest near cut-scores or the mean and much lower for very high- or low-performing students [+W]
- Multiple-choice state tests are prone to ceiling and floor effects and do not measure higher-order skills well [+W]
- Gaining access to state test data involves FERPA constraints and data-request processes typically taking one to six months for review and approval [+W]
- Participation rates for state assessments are generally very high and exceed RCT minimum participation requirements [+W]
- Unreliability in an outcome measure reduces the statistical power of treatment-control comparisons [+W]
Related Research Methods¶
-
Key Sources¶
- May, Henry, Irma Perez-Johnson, Joshua Haimson, Samina Sattar, and Phil Gleason (2009). Using State Tests in Education Experiments: A Discussion of the Issues (NCEE 2009-013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://www.mathematica.org/publications/using-state-tests-in-education-experiments-a-discussion-of-the-issues
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
suitability-and-feasibility-framework-for-deciding-whether-to-use-state-tests-in-education |
title |
Suitability and feasibility framework for deciding whether to use state tests in education experiments |
description |
The paper organizes the decision of whether to use state assessment data in an RCT around two concepts from measurement theory. |
status |
review |
moved_from |
theory/suitability-feasibility-framework-state-tests-rct |
generated |
by process:sweep-kinds on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — may-2009 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.