Rater selection via two-facet Rasch infit filtering substantially increases inter-rater reliability (Krippendorff's alpha) on ICNALE GRA¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Selecting the 12 raters whose infit falls between 0.5 and 1.5 across all aspects raises Krippendorff's alpha substantially relative to all 80 raters. → Stefano Bannò 2026

Evidence¶

Stefano Bannò 2026¶

Stefano Bannò, Kate Knill, Mark Gales. (2026). Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs. https://arxiv.org/abs/2605.04298 doi:10.48550/arxiv.2605.04298

q2 · i? · design · r2

Psychometric calibration analysis in Section 3.1.1 (Table 1): alpha over all 80 raters ranges 0.172-0.297, while the 12 selected raters (18,480 ratings) range 0.409-0.579 across the ten analytic aspects and holistic scores.

"rater selection based on infit statistics leads to a substantial increase in observed reliability."

Discussion¶

Related Claims¶

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it
• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• Coding achieved satisfactory inter-coder reliability and the model reached theoretical saturation — related
• Raters differed significantly in severity/leniency in the MFRM analysis — related