FFL evaluation scores are significantly and positively correlated with principals' contributions to student achievement (value-added)¶

Claim · All claims
Evidence · 2 studies (3 entries) · 2 associational r2 · q2–q3
Currency · newest study 2016 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2019: tool and model accuracy changes quickly

Subclaims¶

q3 i? FFL evaluation scores are significantly and positively correlated with estimates of principals' contributions to student achievement among more than 300 Pennsylvania principals. → Moira McCullough 2016

q2 i? Principals with larger estimated contributions to student achievement growth (value-added) scored higher overall and on multiple components and domains of the FFL. → McCullough 2016

q2 i? Higher total FFL scores and scores in two of the four FFL domains are significantly or marginally significantly associated with value-added in all subjects combined and in math specifically. → McCullough 2016 (2)

Evidence¶

Moira McCullough 2016¶

Moira McCullough, Stephen Lipscomb, Hanley Chiang, Brian Gill. (2016). Do Principals' Professional Practice Ratings Reflect Their Contributions to Student Achievement? Evidence from Pennsylvania's Framework for Leadership. Working Paper 46. Cambridge, MA: Mathematica Policy Research. https://www.mathematica.org/publications/do-principals-professional-practice-ratings-reflect-their-contributions-to-student-achievement

q3 · i? · associational · r2

Correlational validity study of Pennsylvania's Framework for Leadership using data on more than 300 principals; the study "find[s] that FFL evaluation scores are significantly and positively correlated with estimates of principals' contributions to student achievement." No effect size is printed.

"Using data on more than 300 principals, we find that FFL evaluation scores are significantly and positively correlated with estimates of principals' contributions to student achievement."

McCullough 2016¶

McCullough, M., Lipscomb, S., Chiang, H., Gill, B., & Cheban, I. (2016). Measuring School Leaders' Effectiveness: Final Report from a Multiyear Pilot of Pennsylvania's Framework for Leadership. REL Mid-Atlantic, IES. https://ies.ed.gov/ncee/edlabs

q2 · i? · associational · r2

Concurrent validity was assessed through a regression model relating school leaders' estimated contributions to student achievement growth to their FFL scores; higher-FFL principals showed larger value-added.

"Principals with larger estimated contributions to student achievement growth (value-added) scored higher overall and on multiple components and domains than principals with lower estimated contributions."

McCullough 2016 (2)¶

McCullough, M., Lipscomb, S., Chiang, H., Gill, B., & Cheban, I. (2016). Measuring School Leaders' Effectiveness: Final Report from a Multiyear Pilot of Pennsylvania's Framework for Leadership. REL Mid-Atlantic, IES. https://ies.ed.gov/ncee/edlabs

q2 · i? · associational · r2

The regression analyses found "significantly or marginally significantly" higher total FFL scores and two of four domains associated with combined-subject and math value-added.

"Higher total FFL scores and scores in two of the four FFL domains are significantly or marginally significantly associated with both value-added in all subjects combined and value-added in math specifically."

Discussion¶

Related Claims¶

• FFL scores correlate more strongly with contributions to math achievement than to achievement in other subjects — a narrower finding that bears on this claim
• Higher FFL scores associate with larger value-added for middle school principals but not for elementary or high school principals — a narrower finding that bears on this claim
• FFL has concurrent validity primarily among principals with three to six years of tenure as their school's leader — a narrower finding that bears on this claim
• The strongest FFL validity relationships are in the systems leadership and professional and community leadership domains — a narrower finding that bears on this claim
• Mathematica developed and estimated VAMs for use in Phase 1 of the Pennsylvania evaluation pilot — related
• FFL scores were not associated with school leaders' estimated contributions to student achievement growth — reports the opposite
• There is little prior evidence on the validity of principal evaluation measures, and this is the first study linking practice ratings to credible value-added measures — related
• School value added can reflect school-specific influences on student achievement growth that are outside principals' control — related
• Federal policy requirements drove growth modeling and teacher value-added evaluation under NCLB and RTTP — related
• A multiyear pilot study analyzed the score variation, internal consistency, score stability, and concurrent validity of Pennsylvania's Framework for Leadership — related
• The report concludes more validity evidence is needed before using FFL scores to identify effective and ineffective school leaders — reports the opposite
• The five essential supports matter in combination: strength on clusters of supports, not isolated domains, predicts improvement — related
• Teachers with higher Framework for Teaching scores tended to make larger contributions to student achievement (value added) — related
• The study assessed rating stability for 2,424 principals and predictive validity for 123 principals using Pennsylvania statewide data — related