The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation¶
Claim · All claims
Evidence · 2 studies · 2 causalr2·q2·i2–i3
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i2 In a pilot validation on 40 recordings, total-score correlation with human ratings was weak to moderate (r = 0.36), and dimension-level agreement varied widely. → Chen C and Liu T 2026
q2 i3 Logical coherence (r = 0.79) and linguistic accuracy (r = 0.63) correlated strongly with human ratings; information fidelity did not (r = 0.12, non-significant). → Chen C and Liu T 2026 (2)
Evidence¶
Chen C and Liu T 2026¶
Chen C and Liu T (2026) Impact of automated scoring on interpreting performance and self-regulated learning: evidence from a pedagogical experiment. Front. Psychol. 17:1867463. https://doi.org/10.3389/fpsyg.2026.1867463
q2 · i2 · causal · r2
A preliminary criterion check scored 40 pilot recordings with both the Yunyi system and two trained raters; automated scores were generally lower than human scores (M = 53.32 vs. M = 80.40), and "the two sets of total scores had weak to moderate correlation" (r = 0.36).
"Nevertheless, the two sets of total scores had weak to moderate correlation (Pearson r = 0.36, p = 0.022, 95% CI [0.06, 0.60]; Spearman ρ = 0.35, p = 0.026, 95% CI [0.04, 0.60])."
Chen C and Liu T 2026 (2)¶
Chen C and Liu T (2026) Impact of automated scoring on interpreting performance and self-regulated learning: evidence from a pedagogical experiment. Front. Psychol. 17:1867463. https://doi.org/10.3389/fpsyg.2026.1867463
q2 · i3 · causal · r2
Dimension-level agreement on the 40 pilot recordings was strongest for logical coherence (r = 0.79) and linguistic accuracy (r = 0.63), moderate for delivery fluency (r = 0.45), and weakest — non-significant — for information fidelity (r = 0.12), the dimension carrying the greatest rubric weight (35%).
"Logical coherence ( r = 0.79, p < 0.001, 95% CI [0.64, 0.89]) and linguistic accuracy ( r = 0.63, p < 0.001, 95% CI [0.39, 0.78]) showed strong correlations with human ratings; delivery fluency (r = 0.45, p = 0.004, 95% CI [0.16, 0.67]) a moderate one; and information fidelity ( r = 0.12, p = 0.478, 95% CI [ −0.20, 0.41]) a weak, non-significant correlation."
Discussion¶
Related Claims¶
- Gains from automated scoring concentrated in linguistic accuracy and logical coherence but were absent in information fidelity and delivery fluency — related
- In a 16-rater usability survey after a session, CoLearn scores an overall mean of 4.0/5 with no dimension below ≈3.7, and 75% (12/16) would reuse it; Responsiveness is the lowest dimension — related
- AEEGS scores show high absolute agreement with 12 human expert raters' scores on mathematical Economics essay items (average-measures ICC = 0.863) — related
- Each of the 12 human raters' scores correlates significantly with AEEGS scores, with coefficients ranging from 0.604 to 0.864 — related
- Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
- Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related
- LLM scoring performance is dimension-specific: Reasoning shows the strongest agreement and Terminology the weakest across all three strategies — a narrower finding that bears on this claim
- Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — a broader claim this one bears on
- LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions — related
- Grammatical Accuracy is the most stable and objective scoring dimension for LLMs, while Task Response is the most challenging — a narrower finding that bears on this claim
- Students were reasonably accurate overall across the nine scored interview questions (M = 6.17, SD = 2.12), with accuracy varying by question type — related
- Human ratings and automated metrics converge on coherence and temporal alignment as the dimensions most enhanced by the dual gatekeeping pipeline — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
yunyi-human-agreement-varies-by-dimension |
title |
The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation |
description |
The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
source_checked |
2026-10-20 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
2 — chen-c-and-liu-t-2026, chen-c-and-liu-t-2026-2 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.