Evaluate a leadership-evaluation instrument on four measurement properties before high-stakes use¶

Strategy · All strategies
Evidence · 7 claims (6 for, 1 mixed) · 5 studies (4 associational, 1 design), q2–q3 · 0 of 5 report an effect size · 7 claims rest on one study

Description¶

The study's approach was to examine a leader-evaluation tool across four measurement properties—"score variation, internal consistency, score stability, and concurrent validity"—during a multiyear pilot before full implementation. This strategy treats piloting as a period for gathering psychometric evidence about an evaluation instrument. The brief presents this as the study's analytic frame for the FFL pilot.

Design Implications¶

Context¶

Requirements¶

• A multiyear pilot implementation period during which instrument scores are collected

Constraints¶

• The supplied text reports the analysis plan but not the pilot's detailed results

Target Learners¶

• School leaders (principals and other administrators)

Target Learning Goals¶

• Reliable and valid measurement of educator effectiveness

Claims¶

• School leaders' FFL scores are moderately stable year to year (correlation of .54) [+W]
• The FFL shows good internal consistency across its full scale and its four categories of leadership practices [+W]
• The post-study survey instrument showed high internal consistency (alpha = 0.90) [+W]
• The post-interaction reflection instrument showed satisfactory internal consistency (Cronbach's α = .863) [+W]
• Higher FFL scores associate with larger value-added for middle school principals but not for elementary or high school principals [~W]
• Most variation in teacher evaluation scores is within schools rather than between schools [+W]
• FFL has concurrent validity primarily among principals with three to six years of tenure as their school's leader [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Moira McCullough, Stephen Lipscomb, Hanley Chiang, Brian Gill, Irina Cheban. (2016). Measuring School Leaders' Effectiveness: Findings from a Multiyear Pilot of Pennsylvania's Framework for Leadership (Stated Briefly). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, REL Mid-Atlantic. https://ies.ed.gov/ncee/