Statistical item selection that maximizes norm-referenced discrimination undermines criterion-referenced test definitions¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q2
Currency · newest study 1970 · not yet checked against its sources · not reviewed by a person
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on an ERIC document (a report, conference paper or thesis). It is kept because 3 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Screening items for desirable statistical characteristics changes the definitions of the behavioral categories a criterion-referenced test is meant to represent. → Nitko 1970

Evidence¶

Nitko 1970¶

Nitko, Anthony J. (1970). Criterion-Referenced Testing in the Context of Instruction. https://eric.ed.gov/?id=ED047010

q2 · i? · theoretical · r2

Theoretical argument in the section on norm-referenced scores from criterion-referenced tests. The paper argues that since domains of tasks are determined before test construction, statistical item selection is inappropriate: it "will change the definitions of the behavioral categories". No empirical data are reported.

"To screen out some items for inclusion on a particular test because they possess desirable statistical characteristics will change the definitions of the behavioral categories (cf. Osburn, 1968)."

Discussion¶

Related Claims¶

• Criterion-referenced test items should represent skills essential to learning to read and be arranged in the hierarchy used for teaching — related
• Criterion-referenced tests indicate whether a particular skill or objective has been achieved rather than comparing performance to other test takers — related
• Criterion-referenced tests require empirically-based construct validation studies — related
• Intersection point k0 of item difficulty and discriminating curves provides a data-driven item-deletion criterion — related
• Norm-referenced mental-age and IQ metrics distract investigators from the structural and hierarchical aspects of developing abilities — related
• The article argues for shifting from the definition of validity to the concept of effectiveness for classroom assessment score use — related
• Content alignment between a test and its content domain represents key evidence for validating test score inferences — related
• Coaches cautioned that AI-generated feedback should be used formatively rather than for summative assessment because of AI's probabilistic nature — related
• Screening assessments vary in design, administration, and risk definitions such that there is no truly common definition of risk across assessments — related
• Item discrimination screening removed eight items using a critical-ratio criterion above 3 — related
• Critical thinking is rarely defined in HCI research about AI, with fundamental construct and criterion validity challenges — a narrower finding that bears on this claim