Districts face validity, reliability, comparability, and accountability-tension challenges in assessing future-ready skills¶

Claim · All claims
Evidence · 1 study (2 entries) · qualitative r1 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2030: baselines and conditions change

Subclaims¶

q2 i? Districts struggle to ensure validity and reliability of locally designed tools, find scores incomparable across districts, and experience tension with state and national accountability systems. → Kyle Dunbar 2025 q2 i? Locally designed rubrics differing across districts limit score comparability, making it difficult to gauge performance against students elsewhere. → Kyle Dunbar 2025 (2)

Evidence¶

Kyle Dunbar 2025¶

Kyle Dunbar, Katie Wilczak, Kelly Mills, Nick Mireles, and Alison Shell. (2025). Portraits in Progress: Emerging Future-Ready Skills Assessment Strategies. Digital Promise and Pearson. https://digitalpromise.dspacedirect.org/items/0a2c7a9e-634b-4ec1-ba5d-3c8bfa19e696 doi:10.51388/20.500.12265/266

q2 · i? · qualitative · r?

Insight from interviews and convening. One district found a measure defined skills differently than they did, so results did not represent student ability on the district's meaning of the skill; teacher scoring also varied even with the same rubric, and self-reported survey data may lack validity.

"Districts that have Portrait of a Graduate assessments in place expressed a need for guidance, systems, and processes to support the technical quality of the assessment tools they create."

Kyle Dunbar 2025 (2)¶

Kyle Dunbar, Katie Wilczak, Kelly Mills, Nick Mireles, and Alison Shell. (2025). Portraits in Progress: Emerging Future-Ready Skills Assessment Strategies. Digital Promise and Pearson. https://digitalpromise.dspacedirect.org/items/0a2c7a9e-634b-4ec1-ba5d-3c8bfa19e696

q2 · i? · qualitative · r1

Cross-district finding on rubric comparability. The report frames this as a tension: districts value unique and localized skill constructs, measures, and tools, yet also desire comparability and validity, which the report says can be overcome but requires additional research and development.

"These locally-designed rubrics are all slightly different from one another, which limits the comparability of scores across districts, making it difficult for districts to get a sense of how their students are performing compared to students elsewhere in other states or districts."

Discussion¶

Related Claims¶

• Districts use a wide variety of methods to assess Portrait of a Graduate skills, varying across contexts and grade levels — related
• Districts report more success with future-ready skill assessment focused on student growth than on proficiency — related
• Evidence on reliability and validity of alternative student growth measures, especially student learning objectives, is limited — related
• The Content Proximity spring 2022 pilot study examined validity, reliability, and test score comparability of MAP Growth assessments using the new item-selection algorithm — related
• Alternative student growth measures brought financial costs and implementation challenges around teacher time, test administration, and rigor — related
• Post-COVID norm shifts and NWEA's enhanced item selection algorithm change the interpretation of MAP Growth metrics for school accountability — related
• High school GPA is a better predictor of college graduation than standardized test scores, yet districts continue to rely on test-based readiness benchmarks — related
• The authors assert that innovative assessment measures capture POG competencies more fully than traditional standardized tests — related
• Integrating POG assessments into traditional standardized-test-centered accountability systems is complex, and lack of a standardized framework can produce inconsistent measurement across regions — a broader claim this one bears on
• District teams struggled to produce rigorous, reliable evidence for assessing their improvement work — related
• States integrate performance-based assessments as a complement or alternative to traditional multiple-choice tests within a broader assessment strategy — related
• The VCP rubric was difficult for teams to use because it lacked standardized calibration and its process and content categories were unclear — a narrower finding that bears on this claim
• Case-study states rely on locally developed performance assessments created and piloted by teachers, with collaborative scoring — related
• Performance assessments can be reliably scored when designed appropriately and used with other measures — related
• Instruments show great heterogeneity in measurement quality, with validity or reliability information lacking in many cases — related
• An analysis of 69 district Portraits of a Graduate finds six Skill Sets that capture 90 percent of the attributes districts articulate — related
• Contextual conditions beyond district control shape how a college readiness indicator system can be implemented — related
• Using typical measures of school performance may lead state and local decisionmakers astray when identifying schools for support — related
• Districts defined pilot success differently, making it difficult to set a standard for evaluating product effects on student improvement — related
• Standardized test proficiency provides biased information about school quality, indicating a need for alternative metrics — related
• Education researchers have not isolated cost-effective, reliable predictors of teacher performance, motivating exam-based certification — related
• Fewer than one fifth of districts with Portraits have taken the step of identifying actionable Practices within their Skill Sets — related