Two rubric dimensions (claims and evidence, comparison) showed low inter-scorer consistency in the pilot¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2021 · checked against its sources 2026-10-09 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? Scores on the claims and evidence dimension and the comparison dimension were not very consistent across scorers, with further analysis suggesting likely causes for each. → Iwatani 2021

Evidence¶

Iwatani 2021¶

Iwatani, E., Means, B., Seylar, J., and Hardy, A. (2021). Rubrics on historical thinking skills for assignments and student work: Initial validity evidence. Digital Promise. https://doi.org/10.51388/20.500.12265/112

q2 · i? · design · r2

Per-dimension consistency analysis from the pilot scoring session reported in the key findings and Appendices E and F. The article states "ZeUe nRW YeU\ cRnViVWenW acURVV VcRUeUV" for claims and evidence and comparison, and attributes the former to teachers being unaccustomed to assessing claims and evidence together, since AP assesses these separately.

"HRZeYeU, VcRUeV Rn WZR Rf Whe dimenViRnV, ³ClaimV and eYidence´ and ³CRmSaUiVRn,´ ZeUe nRW YeU\ cRnViVWenW acURVV VcRUeUV. FXUWheU anal\ViV VXggeVWed WhaW Whe fRUmeU iV likel\ becaXVe WeacheUV aUe nRW XVed WR aVVeVVing claimV and eYidence WRgeWheU (AP VeSaUaWeV WheVe)"

Discussion¶

Related Claims¶

• Rubrics with parallel sentence structure, unambiguous wording and well-specified dimensions have lower grading variance, and revising rubrics decreased median grading error from 12.4% to 9.9% — the general claim this finding bears on, with the studies behind it
• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• Scorers found all rubric dimensions important, with feedback bearing mainly on the scoring and training process rather than rubric design — related
• World History Project assignments scored higher on most learning opportunity rubric dimensions than other assignments — related
• Six historical thinking dimensions — historical argumentation, causation, comparison, contextualization, continuity and change over time, and sourcing — were selected as rubric assessment targets based on convergence across U.S. national frameworks and the WHP course guide — related
• Rubric scoring supports answering diagnostic questions about opportunities for and evidence of historical thinking — related
• An evaluation of the World History Project detected no statistically significant difference from comparison students in the other five historical thinking skills — related
• World History Project adoption increases learning opportunities for historical thinking in summative activities relative to business-as-usual curricula — related
• Adapting the scoring mechanism substantially reduced ceiling effects found in the pilot study — related
• Empirical progression research is uneven across historical thinking dimensions: most research covers causal reasoning and sourcing, while contextualization and comparison remain thin — related