Rubric-based assessment showed substantial to almost-perfect inter-rater reliability across all three time points¶

Claim · All claims
Evidence · 1 study · associational r1 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Inter-rater reliability (Cohen's κ) was 0.80–0.89 for the conceptual rubric and 0.73–0.83 for the technical rubric. → Francisco José Jiménez-Hornero 2026

Evidence¶

Francisco José Jiménez-Hornero 2026¶

Francisco José Jiménez-Hornero, Eduardo Gutiérrez de Ravé. (2026). Conceptual and procedural knowledge retention in data visualization: a longitudinal study of doctoral learning and instructional design implications. Instructional Science. https://doi.org/10.1007/s11251-026-09808-0

q3 · i? · associational · r1

Two authors independently scored all blinded submissions across T0–T2 after calibration sessions; discrepancies over one point were resolved by discussion. The κ values fall in the substantial to almost-perfect range per Landis and Koch benchmarks.

"Overall, inter-rater reliability was high for both rubrics (Tables 3 and 4), confirming that the scoring process was stable and replicable across time points. The conceptual-under -standing rubric (κ = 0.80–0.89) showed slightly higher and more uniform agreement than the technical-execution rubric (κ = 0.73–0.83), which exhibited greater variability across crite -ria."

Discussion¶

Related Claims¶

• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• The review's coding framework showed very good inter-coder agreement (κ = 0.89, p < 0.001) on a random sample of six papers — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Doctoral students' conceptual understanding of data visualization declined significantly over twelve months following structured instruction — related