District teams struggled to produce rigorous, reliable evidence for assessing their improvement work¶

Claim · All claims
Evidence · 1 study · qualitative r1 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · recheck by 2026: baselines and conditions change

Subclaims¶

q2 i? Many districts found their evidence was not as rigorous and reliable as needed and improved it through expert consultation. → Bugler 2021

Evidence¶

Bugler 2021¶

Bugler, D. (2021). School District Collaboration: Lessons From Districts Participating in California Education Partners' California Language & Learning Initiative (CALLI). WestEd. https://eric.ed.gov/?id=ED616075

q2 · i? · qualitative · r1

Interview data show teams initially relied on state test scores, then interim and classroom assessments after COVID limited statewide testing, and acknowledged "the need to improve the quality of the evidence they use to assess progress." No effect size.

"Many districts found that their evidence was not as rigorous and reliable as it needed to be. The collaboration helped teams develop better evidence through consultation with content experts, other district teams, and Ed Partners staff."

Discussion¶

Related Claims¶

• Districts rely more on peer recommendations and pilots than on rigorous evidence when choosing ed-tech products — related
• A university–district partnership in Multnomah County, Oregon, delivered district-driven research support while practitioner-scholars learned to perform rigorous evaluation work firsthand — related
• Districts face validity, reliability, comparability, and accountability-tension challenges in assessing future-ready skills — related
• Education stakeholders report that limited, hard-to-find, and rarely trustworthy information about edtech products creates significant barriers to selection — related
• The VCP rubric was difficult for teams to use because it lacked standardized calibration and its process and content categories were unclear — related
• University–district research partnership models vary widely and many struggle to develop collaboratively designed research that assists districts and improves the profession — related
• District improvement teams valued CALLI for peer learning, expert access, and structure that built capacity for continuous improvement — related
• Instructional design models are rarely tested against outcomes; their credibility comes from practitioners finding them useful. — related
• No readily accessible sources of rigorous evidence exist for most ed-tech products — related
• Applying the adapted CTA framework in the discharge pilot uncovered previously unknown stakeholders and barriers — related
• Districts defined pilot success differently, making it difficult to set a standard for evaluating product effects on student improvement — related