Grading variability: 40.7% of grade variance lies between teachers and essay quality is the largest random effect¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? 40.7% of total variance in grades was attributed to differences between pre-service teachers. → Lopera-Oquendo 2024 q2 i? Essay quality was the largest source of variability among individuals (τ11 Task = 1.427). → Lopera-Oquendo 2024 (2)

Evidence¶

Lopera-Oquendo 2024¶

Lopera-Oquendo, C., Lipnevich, A. A., & Mañez, I. (2024). Rating writing: Comparison of holistic and analytic grading approaches in pre-service teachers. Learning and Instruction, 94, 101992. https://doi.org/10.1016/j.learninstruc.2024.101992

q2 · i? · causal · r1

Null HLM on standardized grades nested in participants estimated within-grade variance and between-individual variance, both non-zero; "40.7% of the total variance in grades" was between teachers (LRTest = 175.94, p <.001).

"results indicated that 40.7% of the total variance in grades (ICC) was attributed to the differences between pre-service teachers. An ANOVA showed that random effects were statistically significant (LRTest = 175.94, df= 1, p <.001)."

Lopera-Oquendo 2024 (2)¶

Lopera-Oquendo, C., Lipnevich, A. A., & Mañez, I. (2024). Rating writing: Comparison of holistic and analytic grading approaches in pre-service teachers. Learning and Instruction, 94, 101992. https://doi.org/10.1016/j.learninstruc.2024.101992

q2 · i? · causal · r1

In the random-effects part of the final model, the random slope for essay quality (τ11 Task = 1.427) exceeded that for grading method (τ11 Method = 0.056), making essay quality the largest individual-difference source.

"Finally, results also showed that the essay quality (τ11 Task = 1.427) is the largest source of variability among individuals."

Discussion¶

Related Claims¶

• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — related
• Pre-service teachers show high intra-rater consistency between holistic and analytic grades for the same essays — related