Principals rated teaching practice reliably at the low and middle ends of the rating scale, and most principals' severity matched trained observers¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2011 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2014: tool and model accuracy changes quickly

Subclaims¶

q2 i? Principals and observers agreed on Unsatisfactory versus Basic and Basic versus Proficient ratings, and 72 percent of principals rated in line with observers, while 11 percent rated lower and 17 percent higher. → Sartain 2011

Evidence¶

Sartain 2011¶

Sartain, L., Stoelinga, S. R., & Brown, E. R., with Luppescu, S., Kapadia Matsko, K., Miller, F. K., Durwood, C. E., Jiang, J. Y., & Glazer, D. (2011). Rethinking Teacher Evaluation in Chicago: Lessons Learned from Classroom Observations, Principal-Teacher Conferences, and District Implementation. Consortium on Chicago School Research at the University of Chicago. https://consortium.uchicago.edu/publications/rethinking-teacher-evaluation-chicago-lessons-learned-classroom-observations-principal

q2 · i? · associational · r2

A measurement model of rater severity using matched observations from 84 principals and three external observers found most principals "generally in line with the observers," with 11 percent more severe and 17 percent more lenient. The article also reports agreement at the low and middle ends of the scale.

"we determined which principals regularly rated lower than the observers (11 percent of principals), which principals were generally in line with the observers (72 percent of principals), and which principals regularly rated higher than the observers (17 percent of principals)."

Discussion¶

Related Claims¶

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it
• Principals rated some instruction components inconsistently, rating communicating with students, using assessment in instruction, and organizing physical space lower than observers, and student engagement higher — related
• Principals and trained external observers gave aggregate-consistent Danielson Framework ratings in matched observations — possibly the same claim (merge candidate)
• Principals inflated ratings at the high end of the scale, more often identifying instruction as distinguished than observers — related
• Across 78 observed classrooms, 46% of teacher ratings were at the unsatisfactory or basic level, with AMPS teachers earning generally higher ratings than IDS and Renaissance 2010 teachers — related
• Principals were far more likely than trained external observers to rate the same instruction Distinguished rather than Proficient — reports the opposite
• Most evaluators rated consistently, but about 3 percent were muted and 5 percent erratic, and about 10 percent of a sampled subset were extremely severe or lenient — related
• Actual classroom observation practice often differs from best practice, with principal-teacher perception gaps on measures and observer qualifications — related
• Developers of the principal practice instruments provided only partial information about the instruments' reliability and validity — related
• Differences between principal and observer ratings at the high end of the scale were explained largely by teachers' previous evaluation ratings — related