Most evaluators rated consistently, but about 3 percent were muted and 5 percent erratic, and about 10 percent of a sampled subset were extremely severe or lenient¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2017: tool and model accuracy changes quickly

Subclaims¶

q2 i? About 3 percent of principals were muted (awarding nearly the same rating regardless of component difficulty) and about 5 percent were erratic. → Jennie Y. Jiang 2014 q2 i? In a sample of 46 principals, fewer than ten percent were more than half a rating point more severe or lenient than average; specialists were more severe than principals. → Jennie Y. Jiang 2014 (2)

Evidence¶

Jennie Y. Jiang 2014¶

Jennie Y. Jiang, Susan E. Sporte, Stuart Luppescu. (2014). Analytic Memo: Evaluation Data from the First Year of REACH. Consortium on Chicago School Research. https://www.cps.edu/Pages/reachstudents.aspx

q2 · i? · associational · r2

Analysis of all 2012-13 observation data for unusual rating patterns: "About 3 percent of principals were found to be" muted and about 5 percent erratic, with erratic evaluators rating easy components low and hard components high. Most evaluators applied the rubric consistently with predicted component difficulty.

"About 3 percent of principals were found to be "muted," meaning that a principal would award almost the same rating to all teachers, regardless of the component difficulty."

Jennie Y. Jiang 2014 (2)¶

Jennie Y. Jiang, Susan E. Sporte, Stuart Luppescu. (2014). Analytic Memo: Evaluation Data from the First Year of REACH. Consortium on Chicago School Research. https://www.cps.edu/Pages/reachstudents.aspx

q2 · i? · associational · r2

Severity analysis using a randomly selected sample of 46 principals and 16 specialists, with approximately 1,400 observations and 125 joint observations. Fewer than ten percent of sampled evaluators were more than half a rating point more severe or lenient than average; specialists tended to give lower ratings than principals.

"Another set of analyses focused on a randomly selected sample of 46 principals across the district. It utilized independent, joint observations with 16 specialists. Using approximately 1,400 observations and 125 joint observations, we were able to determine how severe or lenient each evaluator (principal and specialist) was."

Discussion¶

Related Claims¶

• Variation across schools in principal-assigned professional practice ratings suggests principals' standards may not be fully consistent across schools — related
• Principals rated teaching practice reliably at the low and middle ends of the rating scale, and most principals' severity matched trained observers — related
• Principals were far more likely than trained external observers to rate the same instruction Distinguished rather than Proficient — related
• Principals inflated ratings at the high end of the scale, more often identifying instruction as distinguished than observers — related
• Principals and trained external observers gave aggregate-consistent Danielson Framework ratings in matched observations — related
• Instructor grading practices vary extremely, with some instructors awarding no honor grades and others awarding honor grades to over half their students — a narrower finding that bears on this claim