Agreement between AI-generated competency scores and human ratings was mixed across studies, with some reporting moderate to strong agreement and others showing score inflation and inconsistent performance¶
Claim · All claims
Evidence · 1 study · reviewr3·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Among studies comparing AI-generated scores with human ratings, findings were mixed: some reported moderate to strong agreement while others showed score inflation and inconsistent performance. → Zhang Wenjia (Stella) 2026
Evidence¶
Zhang Wenjia (Stella) 2026¶
Zhang Wenjia (Stella), Daniels Benjamin B, Mita Carol, Nguyen Hoang, Duong David B. (2026). AI for Clinical Competency Assessment: Scoping Review of Methods and Applications. JMIR Medical Education. https://doi.org/10.2196/92826
q2 · i? · review · r3
Review-level synthesis of the studies' score accuracy and reliability evaluations, charted as part of the review's assessment evidence mapping. The review reports the comparison directionally: "some reported moderate to strong agreement, while others showed score inflation and inconsistent performance". Pooled agreement statistics are not given in the extracted text.
"Among studies comparing AI-generated scores with human ratings, findings were mixed: some reported moderate to strong agreement, while others showed score inflation and inconsistent performance."
Discussion¶
Related Claims¶
- LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — a broader claim this one bears on
- In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — a narrower finding that bears on this claim
- Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
- Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
- Most faculty–GPT-4 scoring discrepancies were only one-point differences, shrinking across iterations — related
- An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
- LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions — related
- History-taking was the most frequently assessed competency in AI-SSP studies (52%), and assessed domains shifted over time toward interactional competencies — related
- Twenty-one studies describing 20 unique AI-SSP systems for clinical competency assessment were published between 2008 and 2026, with 71% published in 2024 or later — a broader claim this one bears on
- LLM-generated essay scores correlated moderately to strongly with expert human assessor scores — a narrower finding that bears on this claim
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.