Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823¶
Claim · All claims
Evidence · 1 study · designr2·q3·i3large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i3 Across three techniques applied to divergent thinking response scoring, correlation with human judges improved from r = 0.781 to r = 0.823 with reduction in error. → Organisciak 2026
Evidence¶
Organisciak 2026¶
Organisciak, P., & Acar, S. (2026). Know when to trust: Making AI scoring more reliable for educational assessment. Behavior Research. https://doi.org/10.3758/s13428-026-03058-1
q3 · i3 · design · r2
Evaluation of three LLM scoring improvements on Alternative Uses Test responses from over 20,000 responses by more than 2,000 participants across multiple studies, reporting "improvements in correlation with human judges (from r = 0.781 to r = 0.823)" and reduced error.
"These methods, tested in divergent thinking response scoring, each show statistically significant positive results, with improvements in correlation with human judges (from r = 0.781 to r = 0.823) and reduction in error."
Discussion¶
Related Claims¶
- Ensemble modeling of multiple LLM scorers yields performance gains in automated divergent thinking scoring — a narrower finding that bears on this claim
- Weighted probabilistic scoring improves LLM-based scoring by considering a broader range of completion possibilities — possibly the same claim (merge candidate)
- Majority-vote ensembling across three runs produced only modest accuracy gains (1% to 5%) with no model showing degradation — a narrower finding that bears on this claim
- Ensembles of operational raters outperform the LLMs: two raters suffice on average across aspects, but larger ensembles are needed for individual aspects — related
- The response-level confidence score correlates positively and significantly with scoring accuracy — related
- Both pedagogy-expertise-weighted and unanimous-vote ensembles frequently worsen LLM alignment with student learning — related
- Model self-confidence gauges LLM confidence in probability estimates and helps identify when machine-generated scoring outputs are trustworthy — a narrower finding that bears on this claim
- Rank-averaged ensembling stabilized predictions but did not outperform the single base model — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-scoring-three-techniques-raise-human-judge-correlation |
title |
Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823 |
description |
Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823 |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — organisciak-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.