LLM observer ratings converge excellently with human expert judgments in applying the CAARS observer instrument to generated narratives¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Human–LLM rating convergence on 20 narratives was excellent (ICC(2,1) = .95), with human inter-rater reliability of .92 and LLM reliability of .87. → Jana Gonnermann-Müller 2026

Evidence¶

Jana Gonnermann-Müller 2026¶

Jana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Thomas Kosch, Sebastian Pokutta. (2026). LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD-Related Persona Specifications. https://arxiv.org/abs/2605.06307 doi:10.48550/arxiv.2605.06307

q2 · i? · causal · r2

Human-rater validation study: five M.Sc.-level psychologists, blind to persona condition, independently rated 20 narratives sampled from the Experiment I workday dataset, compared with three LLM raters. The article states this validates agreement on scoring generated text, not correspondence with lived experience of people with ADHD.

"Human inter-rater reliability was excellent (ICC(2,1) =.92,95%CI[ .85,. 96]), LLM reliability was good (ICC(2,1) =.87,95%CI[.55,.95]), and human–LLM rating convergence was excellent (ICC(2,1) =.95,95%CI[.57,.99])."

Discussion¶

Related Claims¶

• Default personas show a self-report–behavior gap: they endorse moderate-range ADHD characteristics on self-report while displaying only weakly elevated ADHD-like behavior to observers — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• AI grading of the exam is highly stable across five independent runs at the total-score level (ICC(A,1) = 0.967), with lower but still strong cell-level stability (ICC(A,1) = 0.836) — related
• Model family and prompt format moderate the level of simulated ADHD persona expression — related
• Observer-rated behavioral expression of high- and moderate-intensity ADHD personas declines progressively across turns in unscripted LLM conversations — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — a broader claim this one bears on
• Between-conversation persona stability is intensity-dependent: high-intensity ADHD personas cluster tightly while moderate-intensity personas show the highest aggregate variability — related
• In Study 2, inter-rater agreement for rating creative problem solving ideas was good to excellent across indicators — related