Use pilot and field testing to evaluate psychometric complexity, and standardize testing policies for comparability¶

Strategy · All strategies
Evidence · no claims cited

Description¶

Zwick and Mislevy's recommendations to Smarter Balanced and PARCC, quoted in the review, urge designers to "Recognize the tradeoffs between inferential demands and procedural simplicity," to use pilot and field test periods to evaluate whether complex IRT and population models can be simplified, and to standardize testing policies and procedures to ensure data quality. Pilot data should include students at different curriculum points and longitudinal subsamples for growth modeling.

Design Implications¶

Context¶

Requirements¶

• Collection of response data from students at different points in the curriculum and studying material in different orders, plus longitudinal data from at least a subsample of students

Constraints¶

• Data collection at a single occasion will not be sufficient to investigate dimensionality, parameter estimation, and stability issues

Target Learners¶

• K-12 students in large-scale assessment consortia

Target Learning Goals¶

• Valid, comparable summative proficiency scores aggregated across through-year test events

Related Strategies¶

• States should weigh the tradeoff between easily measured indicators (understandable, low data burden) and more complex measures (greater nuance), and explore innovative readiness measures tied to long-term outcomes

Examples¶

-

Key Sources¶

• Garron Gianopulos. (2019). From Through-Course Summative to Adaptive Through-Year Models for Large-scale Assessment: A Literature Review. NWEA. https://www.nwea.org/research/publication/from-through-course-summative-to-adaptive-through-year-models-for-large-scale-assessment-a-literature-review/