Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard¶

Claim · All claims
Evidence · 1 study · associational r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? The three human evaluators showed near-perfect association and excellent absolute agreement (Pearson r = 0.949; ICC(2,1) = 0.949) with negligible error and bias. → Alonso-Carracedo 2026

Evidence¶

Alonso-Carracedo 2026¶

Alonso-Carracedo, M., Fernandez-Boullon, R., Celard, P., Rodríguez-Martínez, F. J., Otero-Cerdeira, L. (2026). Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach. https://arxiv.org/abs/2607.02432 doi:10.48550/arxiv.2607.02432

q3 · i? · associational · r2

Inter-rater reliability analysis of the three instructors' independent blind grades of the 1200 exam responses (Section 4.1.3). The article reports "ICC(2,1) = 0.949(95% CI[0.941,0.955],p < 0.001)", plus Pearson r = 0.949, Weighted Kappa 0.948, MAE 0.028, and Bland-Altman bias of −0.008 points.

"The Intraclass Correlation Coefficient ICC(2,1) = 0.949(95% CI[0.941,0.955],p < 0.001) indicates excellent absolute agreement."

Discussion¶

Related Claims¶

• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• All four evaluated LLMs produce strongly bimodal item-level score distributions on bash exams, and rubric-enhanced prompts amplify the near-perfect-score peak — related
• Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
• Human item-level scores on bash command exams are strongly bimodal, concentrated at zero and near-full credit with little intermediate partial credit — related
• LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions — related
• AI grading of the exam is highly stable across five independent runs at the total-score level (ICC(A,1) = 0.967), with lower but still strong cell-level stability (ICC(A,1) = 0.836) — related
• Expert annotators show high agreement on intervention appropriateness ratings — related
• The TA ground truth was produced under a one-rater-per-item design with co-grading, an exceptions log, instructor audit, and student review, with an estimated grading-error rate of about 0.1% in the student-inspected subsample — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — related
• Expert validation shows high inter-expert reliability (Gwet's AC1 .96 overall) and high expert-cluster agreement (ARI .97, NMI .95) for the general education categorization — related
• Expert review establishes excellent content validity for TAILS (S-CVI/Ave = 0.98; I-CVI 0.86–1.00) — related
• Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus — related
• Automated essay scoring can achieve high agreement with human raters but faces calibration constraints — related
• Scaled qualitative coding of preservice teachers' video-analysis responses reached very good interrater agreement — related
• The review's coding framework showed very good inter-coder agreement (κ = 0.89, p < 0.001) on a random sample of six papers — related
• Disciplinary AI interaction competencies are observable in student response artifacts and scorable with rubrics grounded in student data and expert knowledge — related
• Human inter-rater agreement on the writing rubric is imperfect: no rubric item reaches full agreement and the most discrepant rater generally falls below 80% — reports the opposite
• Experts rated KITE's feedback positively on scaffolding, guidance, coherence, and tone in 93.18% of judgments, with high interrater agreement (κ = 0.88) — related
• Risk of bias among included studies was mixed, with Newcastle–Ottawa ratings ranging from 3 to 9 and high inter-rater agreement
• In Study 2, inter-rater agreement for rating creative problem solving ideas was good to excellent across indicators — related