Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2021 · checked against its sources 2026-10-09 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Intraclass correlation coefficients indicated good overall consistency among three trained scorers applying the activity rubrics (ICC = .74) and student work rubrics (ICC = .71). → Iwatani 2021
Evidence¶
Iwatani 2021¶
Iwatani, E., Means, B., Seylar, J., and Hardy, A. (2021). Rubrics on historical thinking skills for assignments and student work: Initial validity evidence. Digital Promise. https://doi.org/10.51388/20.500.12265/112
q2 · i? · design · r2
Analysis of pilot rubric scores from three expert world history teachers who scored 19 assignments and 54 student work samples after training. As the article reports, "TheUe ZaV gRRd cRnViVWenc\ amRng Whe 3 VcRUeUV RYeUall", with overall ICC = .74 for activity rubrics and ICC = .71 for student work rubrics, characterized as good per Cicchetti (1994) cutoffs printed in the appendices.
"TheUe ZaV gRRd cRnViVWenc\ amRng Whe 3 VcRUeUV RYeUall in aSSl\ing Whe aVVignmenW UXbUicV (ICC = .74) and VWXdenW ZRUk UXbUicV (ICC = .71) (ASSendi[ E & F)."
Discussion¶
Related Claims¶
-
Rubrics with parallel sentence structure, unambiguous wording and well-specified dimensions have lower grading variance, and revising rubrics decreased median grading error from 12.4% to 9.9% — the general claim this finding bears on, with the studies behind it
-
Raters differed significantly in severity/leniency in the MFRM analysis — the general claim this finding bears on, with the studies behind it
- In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
- The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
- The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
- Two rubric dimensions (claims and evidence, comparison) showed low inter-scorer consistency in the pilot — related
- Scorers had difficulty distinguishing rubric scoring from everyday grading and AP scoring, and training took longer than expected — related
- Scorers found all rubric dimensions important, with feedback bearing mainly on the scoring and training process rather than rubric design — related
- World History Project assignments scored higher on most learning opportunity rubric dimensions than other assignments — related
- The historical thinking rubrics were developed with construct and validity evidence from the literature and teacher/reviewer review — related
- Rubric scoring supports answering diagnostic questions about opportunities for and evidence of historical thinking — related
- An evaluation of the World History Project detected no statistically significant difference from comparison students in the other five historical thinking skills — related
- World History Project adoption increases learning opportunities for historical thinking in summative activities relative to business-as-usual curricula — related
- Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
- Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
- Expert validation shows high inter-expert reliability (Gwet's AC1 .96 overall) and high expert-cluster agreement (ARI .97, NMI .95) for the general education categorization — related
- Teachers find World History Project overwhelming in their initial adoption year and make substantial modifications across three main categories — related
- In Study 2, inter-rater agreement for rating creative problem solving ideas was good to excellent across indicators — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
hts-rubrics-good-inter-rater-consistency |
title |
Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency |
description |
Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2021 |
source_checked |
2026-10-09 |
recheck_by |
2024 |
generated |
by process:wiki-ingest on 2026-10-09 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — iwatani-2021 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.