Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-15 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? Making trustworthiness explicit through metrics and visualizations increased inter-rater reliability and helped learning engineers produce more consistent judgments. → Adam Coscia 2026

Evidence¶

Adam Coscia 2026¶

Adam Coscia, Sujata Duwal, Langdon Holmes, Scott Crossley, and Alex Endert. (2026). Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education. https://arxiv.org/abs/2608.04006 doi:10.48550/arxiv.2608.04006

q2 · i? · design · r2

An LLM prompt tournament with 12 learning engineers comparing 5 prompt templates across 30 learner conversations tested whether metrics and visualizations influenced expert decision-making. The article reports that "Making trustworthiness explicit increased inter-rater reliability" though no effect size is printed in the available text.

"Making trustworthiness explicit increased inter-rater reliability while helping learning engineers resolve conflicting objectives and produce more consistent judgments."

Discussion¶

Related Claims¶

• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Trustworthiness metrics and visualizations surfaced previously overlooked pedagogical risks and enabled more deliberate reasoning about trade-offs between competing objectives — related
• Trustworthiness could not be reduced to a fully automated objective; human judgment remains necessary for resolving complex or context-dependent cases — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• Trained scorers applied the historical thinking skills rubrics with good overall inter-rater consistency — related
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
• Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
• CLARA annotations align with blinded educator judgments, with 0.81 pairwise agreement and highest-rated educational interpretability — related
• LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions — related
• Expert annotators show high agreement on intervention appropriateness ratings — a narrower finding that bears on this claim
• Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — related
• Model self-confidence gauges LLM confidence in probability estimates and helps identify when machine-generated scoring outputs are trustworthy — related
• A graded likelihood detector prompt produced more consistent separation than a binary yes/no detector prompt — related
• LLM observer ratings converge excellently with human expert judgments in applying the CAARS observer instrument to generated narratives — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Experts rated KITE's feedback positively on scaffolding, guidance, coherence, and tone in 93.18% of judgments, with high interrater agreement (κ = 0.88) — related
• Rater selection via two-facet Rasch infit filtering substantially increases inter-rater reliability (Krippendorff's alpha) on ICNALE GRA — related
• Expert validation shows high inter-expert reliability (Gwet's AC1 .96 overall) and high expert-cluster agreement (ARI .97, NMI .95) for the general education categorization — related
• Expert raters rate TeachingCoach higher than a GPT-4o baseline on clarity, tone, reflection, and acknowledgment of user input — related
• Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria — related
• Exemplar-based calibration is expected to align student ratings with expert ratings at the criterion level — related
• Categorizing claims on a website required students to read and interpret content beyond what the visualization tool provides, including evaluating trustworthiness — related
• Disciplinary AI interaction competencies are observable in student response artifacts and scorable with rubrics grounded in student data and expert knowledge — related
• LLMs are more intrusive and more consistent than humans, eager to view neutral discussions as opportunities for positive reinforcement — related
• Expert evaluation refined the draft heuristics around language clarity, usability-accessibility separation, and meaningful data interpretation — related
• Directly prompted GPT-4o surface-level corrective edits were judged necessary and effective in 96.14% of 2049 identified edits — a narrower finding that bears on this claim
• Performance assessments can be reliably scored when designed appropriately and used with other measures — related
• Global-judgment criteria of counselor effectiveness lack clarity, making replication fruitless and the research unproductive — related