The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly

Subclaims¶

q2 i? Scoring of physics questions with the framework achieved an overall percentage agreement of 79% and inter-rater reliability (Gwet's AC1) of 81%, which the authors interpret as high consistency. → Zakani 2019

Evidence¶

Zakani 2019¶

Zakani, S., Kaupp, J., Turner, R. D., & Frank, B. (2019). Analyzing implicit science and math outcomes in engineering and technology programs. The Canadian Journal for the Scholarship of Teaching and Learning, 10(1). https://doi.org/10.5206/cjsotl-rcacea.2019.1.7994

q2 · i? · design · r2

Three trained content specialists scored anonymized physics exam questions from a mix of institutions on the first four framework dimensions. Reliability was assessed using percentage of exact agreement and Gwet's AC1, chosen to account for chance agreement and the number of levels within each dimension.

"With an overall percentage agreement of 79% and inter -rater reliability of 81% in scoring physics questions, the framework was shown to be highly consistent."

Discussion¶

Related Claims¶

• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
• The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement — related
• Technology programs examine discipline-related physics topics at higher cognitive levels within a disciplinary context — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• Teaching style scales show moderate inter-scale correlations (.50-.70) and observer agreement on ratings between .50 and .60 — related
• The GED essay is scored holistically on a six-point scale by two independent readers — related
• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• Confidence-aware selective test-time scoring achieves the best average agreement with expert rubric scoring across six NGSS drawing items — related
• Disciplinary AI interaction competencies are observable in student response artifacts and scorable with rubrics grounded in student data and expert knowledge — a broader claim this one bears on
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Expert validation shows high inter-expert reliability (Gwet's AC1 .96 overall) and high expert-cluster agreement (ARI .97, NMI .95) for the general education categorization — related
• Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Experts rated KITE's feedback positively on scaffolding, guidance, coherence, and tone in 93.18% of judgments, with high interrater agreement (κ = 0.88) — related