LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Adding the LLM as an additional rater did not reduce overall inter-rater agreement (alpha .637 vs .639), and LLM scores deviated less from the human mean than human scores deviated from each other (MAD 8.79 vs 10.25). → Dawei Xie 2026

Evidence¶

Dawei Xie 2026¶

Dawei Xie, Khalil Anderson, Tochukwu Eze, Chenghong Lin, Bookyung Shin, and Marcelo Worsley. (2026). CLARA: An AI-Augmented Analytics Dashboard for Collaboration Literacy. https://arxiv.org/abs/2605.17259 doi:10.48550/arxiv.2605.17259

q2 · i? · design · r2

Score-level agreement study comparing LLM and human expert 7C assessments of 10 discussions, each scored 0-100 per dimension by 2-3 independent raters. The article reports "overall MAD:8.79vs.10.25on the 100-point scale" favoring the LLM, and overall Krippendorff's alpha essentially unchanged when the LLM was added (.637 vs .639).

"In absolute terms, LLM-produced scores deviated less from the human mean than individual human scores deviated from each other (overall MAD:8.79vs.10.25on the 100-point scale)."

Dawei Xie 2026 (2)¶

Dawei Xie, Khalil Anderson, Tochukwu Eze, Chenghong Lin, Bookyung Shin, and Marcelo Worsley. (2026). CLARA: An AI-Augmented Analytics Dashboard for Collaboration Literacy. https://arxiv.org/abs/2605.17259

q2 · i? · design · r2

Spearman correlation between LLM scores and human mean scores across the 10 discussions and 70 dimension-level assessments; the article reports "overallρ=.701,p < .001", with Conflict (.765), Communication (.760), and Climate (.711) strongest and Context weakest (.259).

"LLM-produced scores’ rank-ordering of discussions tracked human consensus on most dimensions(overallρ=.701,p < .001)."

Discussion¶

Related Claims¶

• LLM and human written 7C analyses show no overall difference in behavioral alignment or evidence correspondence, but align less on Communication and Constructive dimensions — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks — related
• Rank-based metrics can be misleading for analytic assessment: systems prompted for holistic scoring achieve strong, sometimes stronger, correlations with analytic scores than when prompted for those aspects — related
• Differences between human and ChatGPT feedback quality were small, with means within one point and partial eta-squared ranging from .01 to .12 — related
• Expert rankings validate the Ratio variables, not absolute frequencies, as meaningful proxies for productive dialogue — related
• AI agent comment patterns include validation (22%) at rates comparable to human peer learning communities, with validation-before-extension sequences structurally resembling collaborative knowledge building — related