Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality¶
Claim · All claims
Evidence · 1 study · causalr1·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Rubric-component agreement across raters was slight (Fleiss' Kappa .126 strong essay, .193 weak essay), with roughly 25–27% raw agreement. → Lopera-Oquendo 2024
Evidence¶
Lopera-Oquendo 2024¶
Lopera-Oquendo, C., Lipnevich, A. A., & Mañez, I. (2024). Rating writing: Comparison of holistic and analytic grading approaches in pre-service teachers. Learning and Instruction, 94, 101992. https://doi.org/10.1016/j.learninstruc.2024.101992
q2 · i? · causal · r1
Inter-rater analysis of rubric-component scores from 231 pre-service teachers grading two essays showed "slight agreement between participants", with percentage of agreement of 26.84 (strong) and 24.68 (weak).
"Regarding the consistency among the rubric components, the Fleiss’ Kappa suggested a slight agreement between participants (.126 and .193 for the strong and weak essays"
Discussion¶
Related Claims¶
- Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria — possibly the same claim (merge candidate)
- Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
- Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — reports the opposite
- Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — reports the opposite
- Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus — reports the opposite
- An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
- Grading variability: 40.7% of grade variance lies between teachers and essay quality is the largest random effect — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
low-inter-rater-agreement-preservice-grading |
title |
Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality |
description |
Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2024 |
generated |
by process:wiki-ingest on 2026-10-23 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — lopera-oquendo-2024 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.