Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Human inter-rater reliability was lowest for Systems Thinking Practices (κ = 0.51), and the LLM's lower agreement on STP mirrored this, indicating the variance stems from construct complexity rather than model deficiency. → Sean Savage 2026

Evidence¶

Sean Savage 2026¶

Sean Savage, Anand Shanker, Grace Michlitsch, and N. Sanjay Rebello. (2026). Using LLMs to Detect Growth in Computational Thinking in Introductory Physics. arXiv preprint. https://arxiv.org/abs/2608.06200 doi:10.48550/arxiv.2608.06200

q2 · i? · design · r2

Human inter-rater reliability measured with Fleiss' κ among three researchers coding 50 student responses; STP yielded the lowest agreement of the five metrics. The authors interpret this as reflecting the cognitive demands of interpreting conceptual framing of interacting system components.

"Systems Thinking Practices yielded the lowest agreement (κ= 0.51), though still within the acceptable range [35]. Because STP requires interpreting a student's conceptual framing of interacting sys- tem components, it is inherently the most cognitively de- manding construct to code."

Discussion¶

Related Claims¶

• Rubrics with parallel sentence structure, unambiguous wording and well-specified dimensions have lower grading variance, and revising rubrics decreased median grading error from 12.4% to 9.9% — the general claim this finding bears on, with the studies behind it
• Students in a computationally intensive introductory physics course show large-magnitude growth in Data Practices and moderate growth in Computational Problem-Solving and Systems Thinking on written CT assessments — related
• A multimodal LLM achieves moderate-to-substantial agreement with human consensus on well-defined CT practices but lower agreement on multi-component constructs like Physics Correctness and Systems Thinking — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• The interview coding achieved high inter-rater reliability, with Cohen's Kappa of 0.82 between independent coders — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• LLM-based transcript classifications agreed highly with human judgment for explicit questions (κ = .778) and math talk (κ = .722), but only moderately for need for help (κ = .562) and confusion (κ = .531) — related
• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• Inter-model agreement on LLM skill tagging is moderate (Cohen's kappa 0.51), with strong agreement on S5 and disagreement concentrated in the S2/S3/S4 boundary — related
• The emergent LLM coding pipeline reaches substantial agreement with expert human coding (Cohen's κ = 0.78, raw agreement 84%) on a stratified 10% sample — related
• Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria — related
• Think aloud coding achieved inter-rater reliability of k = .53-.65 (kmax = .81-.82) between two coders using ELAN software — related
• The review's coding framework showed very good inter-coder agreement (κ = 0.89, p < 0.001) on a random sample of six papers — related
• High agreement between two LLM screeners indicates consistency among outputs, not accuracy, because models may exhibit correlated failure patterns — related
• Rubric-based assessment showed substantial to almost-perfect inter-rater reliability across all three time points — related
• Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — related
• LLMs are more intrusive and more consistent than humans, eager to view neutral discussions as opportunities for positive reinforcement — related
• Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus — related
• The objections stage is the most difficult stage of CRITO because it demands fallibilism about one's own argument — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• Revising the rubric to operationalize pedagogical constructs raises human–machine agreement for openness and depth — related
• Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency — related
• The framework treats systems thinking and computational thinking as co-equal, separate entities within the context of modeling — related
• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — related