Weighted probabilistic scoring improves LLM-based scoring by considering a broader range of completion possibilities¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Weighted probabilistic scoring considers a broader range of completion possibilities in deriving a final score and shows statistically significant positive results. → Organisciak 2026

Evidence¶

Organisciak 2026¶

Organisciak, P., & Acar, S. (2026). Know when to trust: Making AI scoring more reliable for educational assessment. Behavior Research. https://doi.org/10.3758/s13428-026-03058-1

q3 · i? · design · r2

Second of the three evaluated techniques, tested in divergent thinking response scoring in the study of over 20,000 Alternative Uses Test responses; the article reports statistically significant positive results without a technique-specific effect size.

"Weighted probabilistic scoring considers a broader range of completion possibilities in deriving a final score."

Discussion¶

Related Claims¶

• Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823 — possibly the same claim (merge candidate)
• Ensemble modeling of multiple LLM scorers yields performance gains in automated divergent thinking scoring — related