Criterion-referenced tests require empirically-based construct validation studies¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q1
Currency · newest study 1970 · not yet checked against its sources · not reviewed by a person

Subclaims¶

Reading the evidence codes

q is the design tier, i the impact, n the sample, and the kind with its r the rigour judged within that kind: what each code means.

This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.

q1 i? Empirical evidence is needed to support contentions that classes of test tasks reflect the behavior or competence of interest, requiring construct validation studies like those recommended for psychological tests. → Nitko 1970

Evidence¶

Nitko 1970¶

Nitko, Anthony J. (1970). Criterion-Referenced Testing in the Context of Instruction. https://eric.ed.gov/?id=ED047010

q1 · i? · theoretical · r2

The paper's 'Need for a Data Base' section argues, as a theoretical position, that criterion-referenced test builders must conduct "many of the same kinds of construct validation studies" recommended for psychological tests, citing Cronbach and Meehl. No validation study is reported in the paper itself.

"Given that the classes of behavior have been defined, empirical evidence is needed to support any contentions that the classes of test tasks do indeed reflect the behavior or competence of interest."

Studies bearing on this claim¶

Does a pass on the check mean what the criterion says: are the items, cut scores and raters dependable? This table gathers the studies behind this claim and the 17 narrower claims that bear on it: 14 distinct studies (6 design, 3 theoretical, 3 associational, 1 causal, 1 qualitative), q1–q3, 0 of 14 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Study	Design	Effect as recorded	Bears on this claim through
Nitko 1970	q1 / q2 · theoretical · r2	i?	this claim; Statistical item selection that maximizes norm-referenced discrimination undermines criter (supports)
Wolkowitz 2024	q1 / q2 · design · r2	i?	DC estimates tend to be lowest when the cut score is set at the peak of the score distribu (supports); Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making (qualifies); The LL-DC, PS-DC, and W-DC methods produce similar decision consistency estimates (within (qualifies)
Gentile 1971	q1 · theoretical · r1	i?	Grade cutoffs are arbitrary because adjacent scores receiving different grades are not sig (supports)
Olson 2002	q2 · design · r2	i?	A body of work standard setting process produced classification agreement rates ranging fr (supports); A mean-classification cut-score method outperformed a contrasting-groups-style mean method (qualifies)
Pikulski 1973	q2 · theoretical · r?	i?	Criterion-referenced tests indicate whether a particular skill or objective has been achie (qualifies)
Ilhan 2018	q3 · associational · r2	i?	Raters differed significantly in severity/leniency in the MFRM analysis (supports)
Morzinski 2010	q2 · design · r2	i?	In one community-college lab class, six scorers rating six teams with an engineering rubri (supports)
Praveen Pathak 2026	q2 · causal · r1	i?	Focused rubric refinement that made required physics and scoring conditions explicit reduc (qualifies)
Weiss 1980	q2 · design · r?	i?	An adaptive mastery testing strategy reduces test length by 30% to 81% while reaching the (qualifies)
Shannon 2026	q2 · associational · r2	i?	The ELA placement test was most precise for students whose scores would place them into de (supports)
Jiayi Wang 2025	q3 · design · r2	i?	Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM c (supports)
Perry 2019	q2 · qualitative · r?	i?	The VCP rubric was difficult for teams to use because it lacked standardized calibration a (supports)
Sartain 2011	q2 · associational · r2	i?	Principals rated teaching practice reliably at the low and middle ends of the rating scale (qualifies)
Stefano Bannò 2026	q2 · design · r2	i?	Rater selection via two-facet Rasch infit filtering substantially increases inter-rater re (qualifies)

Discussion¶

Criterion-referenced test items should represent skills essential to learning to read and be arranged in the hierarchy used for teaching — related
Criterion-referenced tests indicate whether a particular skill or objective has been achieved rather than comparing performance to other test takers — related
Statistical item selection that maximizes norm-referenced discrimination undermines criterion-referenced test definitions — related
The five dimensions of the developed thinking styles scale show structural validity via confirmatory factor analysis — a narrower finding that bears on this claim
Critical thinking is rarely defined in HCI research about AI, with fundamental construct and criterion validity challenges — a narrower finding that bears on this claim

Page metadata

Field	Value
`type`	`claim`
`id`	`criterion-referenced-tests-need-construct-validation`
`title`	`Criterion-referenced tests require empirically-based construct validation studies`
`description`	`Criterion-referenced tests require empirically-based construct validation studies`
`status`	`review`
`evidence_strength`	`weak`
`newest_evidence`	`1970`
`generated`	by `process:wiki-ingest` on `2026-09-25`
`trust`	unverified — no one has confirmed this page's content
`sources`	1 — `nitko-1970`

Frontmatter is the page's machine-readable record — see the Schema & Guide.