Single-initiative data analysis has not always changed agency-wide policies and practices, and many staff may lack skills to interpret findings¶

Claim · All claims
Evidence · 1 study (2 entries) · qualitative r? · q2
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2019: baselines and conditions change

Subclaims¶

q2 i? Although agencies analyze data for single initiatives, this measurement and analysis has not always impacted agency-wide policies and practices, according to the staff and fellows interviewed. → Cassie Pickens Jewell 2014 q2 i? It appears likely that many agency staff other than fellows do not know what questions to ask, what data to use, or how to interpret findings and draw inferences to improve policy and practice. → Cassie Pickens Jewell 2014 (2)

Evidence¶

Cassie Pickens Jewell 2014¶

Cassie Pickens Jewell, Libby Makowsky, Kristin Hallgren, Jacob Hartog. (2014). Changing Education Agencies from the Inside Out: Year 2 Report on the Strategic Data Project and Education Pioneers. Princeton, NJ: Mathematica Policy Research. https://www.mathematica.org/publications/changing-education-agencies-from-the-inside-out-year-2-report-on-the-strategic-data-project

q2 · i? · qualitative · r?

Key finding from the implementation study's interviews with agency staff and fellows. The report attributes the limited agency-wide impact to the initiative-specific nature of the analysis, per informant reports. Descriptive; no effect sizes.

"Although agencies analyze data for single initiatives, this kind of measurement and analysis has not always impacted agency-wide policies and practices, according to the staff and fellows from whom we gathered information."

Cassie Pickens Jewell 2014 (2)¶

Cassie Pickens Jewell, Libby Makowsky, Kristin Hallgren, Jacob Hartog. (2014). Changing Education Agencies from the Inside Out: Year 2 Report on the Strategic Data Project and Education Pioneers. Princeton, NJ: Mathematica Policy Research. https://www.mathematica.org/publications/changing-education-agencies-from-the-inside-out-year-2-report-on-the-strategic-data-project

q2 · i? · qualitative · r?

Authors' interpretation from the case-study interviews, hedged with "It appears likely": staff other than fellows may lack skills in question formulation, data selection, and interpretation for policy and practice. No effect size is reported.

"It appears likely that many agency staff (other than fellows) do not know what questions to ask, what data to use, or most importantly, how to interpret findings and draw inferences to improve policy and practice."

Discussion¶

Related Claims¶

• Agency experiences with data practices were documented through interviews covering collection, use, analysis, and reporting — related
• SDP and EP partnerships enabled education agencies to develop new data-use practices that agencies continue to refine — related
• Capacity for data use was similar among SDP- and EP-partnered agencies although the two programs' approaches differ — related
• The SDP and EP programs aim to enhance education agencies' data capacity through support, training, and placement of additional staff — a broader claim this one bears on
• The NIH Intramural Research Loan Repayment Program evaluation combined an outcomes evaluation with a process analysis drawing on focus groups, staff interviews, and administrative data — related
• Survey findings reflect intended plans that may differ from eventual implementation, and the survey structure limits what can be inferred — related
• Agencies built multi-source linked data systems and worked to increase analytic capacity through hiring, vendor partnerships, and training — related
• Many low-achieving students were initially unable to interpret KCA data productively when first using the tool. — related
• Agency data-use profiles were developed from fall and winter 2013–14 site visit interviews — related
• Districts with data systems in place still face challenges in the strategic use of data — related