Year-averaged classroom observation scores provided only modest reliability for identifying teachers needing support, and dimension-level differences were too unreliable to signal specific areas for improvement¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2017 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2020: tool and model accuracy changes quickly

Subclaims¶

q2 i? Four-window average overall scores had Year 2 reliabilities of .53 to .61 (CLASS) and .70 to .77 (FFT); single observations had reliabilities of .33 (CLASS) and .51 (FFT). → Garet 2017 q2 i? Differences across dimensions had limited reliability even when averaged over the year (.35 to .43 for CLASS and .18 to .30 for FFT in Year 2). → Garet 2017 (2)

Evidence¶

Garet 2017¶

Garet, M.S., Wayne, A.J., Brown, S., Rickles, J., Song, M., and Manzeske, D. (2017). The Impact of Providing Performance Feedback to Teachers and Principals, Executive Summary (NCEE 2018-4000). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. http://ies.ed.gov/ncee

q2 · i? · causal · r2

Psychometric analysis of observation ratings from the study's four annual observation windows. The article notes 39 to 47 percent of variation was measurement error for CLASS, and that single-observation reliability was .33 for CLASS and .51 for FFT.

"In Year 2, for example, depending on the assumptions used, reliability estimates for the four-window average overall scores were between .53 and .61 for the CLASS."

Garet 2017 (2)¶

Garet, M.S., Wayne, A.J., Brown, S., Rickles, J., Song, M., and Manzeske, D. (2017). The Impact of Providing Performance Feedback to Teachers and Principals, Executive Summary (NCEE 2018-4000). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. http://ies.ed.gov/ncee

q2 · i? · causal · r2

Dimension-level reliability analysis of the two observation rubrics. The article concludes the measures "provided limited information to indicate the areas of practice educators most needed to improve".

"While most teachers received ratings that differed across dimensions, the differences were not sufficiently reliable to identify dimensions for improvement, even when averaged over the year (.35 to .43 for the CLASS and .18 to .30 for the FFT in Year 2)."

Discussion¶

Related Claims¶

• A two-year performance feedback intervention improved teachers' classroom practice on one of two observation rubrics (50th to 57th percentile) but had no impact on practice as measured by the other rubric — related
• Year-averaged classroom observation scores provided reliable information to distinguish overall teacher performance (reliabilities .53–.61 and .70–.77), but dimension-level differences had limited reliability (.35–.43 and .18–.30) — possibly the same claim (merge candidate)