Districts defined pilot success differently, making it difficult to set a standard for evaluating product effects on student improvement¶

Claim · All claims
Evidence · 1 study · qualitative r2 · q2
Currency · newest study 2015 · checked against its sources 2026-10-18 (automated, 1 of 1 entry) · not reviewed by a person
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because 3 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Varied definitions of pilot success across districts complicate evaluating product effects on student improvement within and across districts. → Adams-Bass 2015

Evidence¶

Adams-Bass 2015¶

Adams-Bass, V., Atchison, D., & Moore, L. (2015). Piloting Ed-tech Products in K-12 Public Schools: Pilot-to-Purchase Project. A Report from the University of California Davis School of Education to Digital Promise. https://digitalpromise.dspacedirect.org/items/38af8409-82b3-4f0e-baff-2a093963bada

q2 · i? · qualitative · r2

Qualitative analysis found that while all districts intended to use pilot data to evaluate product effectiveness, the types of data emphasized and analyses varied widely; "Districts also defined success differently," complicating standard evaluation.

"Districts also defined success differently; with varied definitions of success, it is difficult to determine an accurate standard for evaluating the effect of products on student improvement, both within and across districts."

Discussion¶

Related Claims¶

• US federal adult education funding structures (AEFLA and Title I) are fragmented and lack purchasing leverage, making scale difficult for ed-tech entrepreneurs — related
• School districts and ed-tech developers hold different perspectives on the role and value of school-based technology pilots in procurement — related
• District-community Core Team participants across four pilot districts perceived their co-designed solutions successfully met the team-defined intended outcomes — related
• Districts rely more on peer recommendations and pilots than on rigorous evidence when choosing ed-tech products — related
• Instructional designers define dispositions in varied and divergent ways, with several conflating them with soft skills — related
• No readily accessible sources of rigorous evidence exist for most ed-tech products — related
• Districts face validity, reliability, comparability, and accountability-tension challenges in assessing future-ready skills — related
• The VCP rubric was difficult for teams to use because it lacked standardized calibration and its process and content categories were unclear — related
• District teams struggled to produce rigorous, reliable evidence for assessing their improvement work — related
• Screening assessments vary in design, administration, and risk definitions such that there is no truly common definition of risk across assessments — related