A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88)¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Item-level agreement between autorater and expert raters on 0-2 scores was Cohen's Kappa 0.66, described as good agreement. → Globerson 2026 q2 i3 Autorater total scores correlated with human expert total scores at Pearson's r = 0.88 on held-out student submissions. → Globerson 2026 (2)

Evidence¶

Globerson 2026¶

Globerson, A., Keeling, A., Choudhury, A., et al. (2026). Towards Scalable Measurement of Durable Skills. https://arxiv.org/abs/2609.15864 doi:10.48550/arxiv.2609.15864

q2 · i? · design · r2

Lab study with OpenMic: 280 students designed a news segment from a story; 100 submissions refined the prompt and rubrics and 180 evaluated autorater accuracy against trained experts. Item-level "Cohen’s Kappa was0.66", an agreement statistic the article reports without a standardized effect size.

"At the specific item0−2scores, the agreement between the scores of the autorater and those of the human expert raters as measured using Cohen’s Kappa was0.66, corresponding to good agreement [28]."

Globerson 2026 (2)¶

Globerson, A., Keeling, A., Choudhury, A., et al. (2026). Towards Scalable Measurement of Durable Skills. https://arxiv.org/abs/2609.15864. doi:10.48550/arxiv.2609.15864

q2 · i3 · design · r2

Same OpenMic lab study, evaluated on 180 held-out submissions: autorater overall scores vs human expert scores gave "Pearson’s correlation of0.88" (r = 0.88), a large correlation; Figure 11 shows the scatter plot of the two raters' scores.

"A comparison of the autorater overall scores and those of the human experts reveals an extremely high Pearson’s correlation of0.88."

Discussion¶

Related Claims¶

• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• Claude showed the highest alignment with human BREQ responses, and interview-containing prompts aligned better than baseline prompts — related
• An LLM-based analogy judge validated against expert judgments shows moderate-to-strong agreement and screens most generated analogies as meeting baseline adequacy — related
• Confidence-aware selective test-time scoring achieves the best average agreement with expert rubric scoring across six NGSS drawing items — related
• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related
• LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction — related
• Rank-based metrics can be misleading for analytic assessment: systems prompted for holistic scoring achieve strong, sometimes stronger, correlations with analytic scores than when prompted for those aspects — related
• Exemplar-based calibration is expected to align student ratings with expert ratings at the criterion level — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Experts rated KITE's feedback positively on scaffolding, guidance, coherence, and tone in 93.18% of judgments, with high interrater agreement (κ = 0.88) — related