Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Three independent teacher raters reached mean Cohen's linear weighted kappa of 0.78–0.84 and mean PoA above 75% across all six FRISCO dimensions. → Firdausi 2026

Evidence¶

Firdausi 2026¶

Firdausi, H.; Wasis; Sholikah, R. W.; Lemantara, J.; Rosyadi, F. D.; Ginardi, R. V. H.; Pradityo, M. H. A.; Hakim, A. R. (2026). Educational Innovation through Automated Essay Scoring: A Multidimensional Framework for Evaluating Critical Thinking in High School Physics Essays. International Journal of Intelligent Engineering and Systems, Vol.19, No.8. https://doi.org/10.22266/ijies2026.0831.09

q2 · i? · design · r2

Reliability analysis of essay scoring by three high school physics teachers of 106 eleventh-grade responses (Methodology §3.1, Table 2). Disagreements over one point were resolved by discussion with a third rater as arbiter; final labels were mean scores rounded to integers.

"all six FRISCO dimensions yielded mean PoA values above 75%, satisfying the reliability threshold [24] and mean kappa values ranging from 0.78 to 0.84, indicating substantial to almost perfect agreement"

Discussion¶

Related Claims¶

• The interview coding achieved high inter-rater reliability, with Cohen's Kappa of 0.82 between independent coders — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• The review's evidence base comprises 42 peer-reviewed publications selected from 922 records with high inter-rater agreement (Cohen's kappa = 0.88) — related
• The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement — related
• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Two human graders reached almost perfect agreement (QWK = 0.95), though one waterfall-model question showed only substantial agreement — related
• Automated essay scoring can achieve high agreement with human raters but faces calibration constraints — related
• Think aloud coding achieved inter-rater reliability of k = .53-.65 (kmax = .81-.82) between two coders using ELAN software — related
• The review's coding framework showed very good inter-coder agreement (κ = 0.89, p < 0.001) on a random sample of six papers — related
• The best AES models per FRISCO dimension reached strong agreement for Situation (QWK 0.728) and Clarity (QWK 0.763) but only fair agreement for Focus, Reason, and Inference in Indonesian physics essays — related
• AI assessors show emerging reliability: acceptable AI-human rater agreement (Cohen's 0.78) in a 12-educator case study and human-comparable feedback helpfulness ratings with 45 learners — related
• Rubric-based assessment showed substantial to almost-perfect inter-rater reliability across all three time points — related
• GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related
• The emergent LLM coding pipeline reaches substantial agreement with expert human coding (Cohen's κ = 0.78, raw agreement 84%) on a stratified 10% sample — related
• Raters scored significantly higher than AI on Emotional Support and significantly lower than AI on Instructional Support, with no difference for Classroom Organization — related
• Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus — related
• Scaled qualitative coding of preservice teachers' video-analysis responses reached very good interrater agreement — related
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency — related
• Experts rated KITE's feedback positively on scaffolding, guidance, coherence, and tone in 93.18% of judgments, with high interrater agreement (κ = 0.88) — related
• Risk of bias among included studies was mixed, with Newcastle–Ottawa ratings ranging from 3 to 9 and high inter-rater agreement
• In Study 2, inter-rater agreement for rating creative problem solving ideas was good to excellent across indicators — related
• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — reports the opposite