Ensemble modeling of multiple LLM scorers yields performance gains in automated divergent thinking scoring¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Ensemble models assess and yield performance gains from combining multiple models in automated scoring. → Organisciak 2026

Evidence¶

Organisciak 2026¶

Organisciak, P., & Acar, S. (2026). Know when to trust: Making AI scoring more reliable for educational assessment. Behavior Research. https://doi.org/10.3758/s13428-026-03058-1

q3 · i? · design · r2

Third evaluated technique, tested in divergent thinking response scoring; the article reports it among the methods showing statistically significant positive results, with no separate effect size printed for ensembles alone.

"The final technique, ensemble models, assesses the performance gains from combining multiple models."

Discussion¶

Related Claims¶

• Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823 — a broader claim this one bears on
• Rank-averaged ensembling stabilized predictions but did not outperform the single base model — reports the opposite
• A stacking ensemble of LG, SVM, NB, KNN and XGB achieved the best fine-tuning F1-score (0.88), outperforming individual models such as SVM (0.86) — a broader claim this one bears on
• Both pedagogy-expertise-weighted and unanimous-vote ensembles frequently worsen LLM alignment with student learning — reports the opposite
• Majority-vote ensembling across three runs produced only modest accuracy gains (1% to 5%) with no model showing degradation — a narrower finding that bears on this claim
• Ensembles of operational raters outperform the LLMs: two raters suffice on average across aspects, but larger ensembles are needed for individual aspects — related
• Combining human and LLM-generated responses at a 1:1 ratio yields more accurate educational item calibration — related
• Weighted probabilistic scoring improves LLM-based scoring by considering a broader range of completion possibilities — related
• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects — related