Ocsai and confidence Python libraries for confidence-aware LLM scoring¶
Product or Programme · All products and programmes
Evidence · 4 claims (3 for, 1 mixed) · 1 study (1 design),q3· 1 of 1 report an effect size · 4 claims rest on one study
Description¶
Python libraries released by Organisciak et al. for confidence measurement and weighted probabilistic scoring of open-ended text using OpenAI, Gemini, or HuggingFace models.
Components¶
Claims¶
- Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823 [+W]
- Model self-confidence gauges LLM confidence in probability estimates and helps identify when machine-generated scoring outputs are trustworthy [+W]
- Weighted probabilistic scoring improves LLM-based scoring by considering a broader range of completion possibilities [+W]
- Ensemble modeling of multiple LLM scorers yields performance gains in automated divergent thinking scoring [~W]
Related Products and Programmes¶
Key Sources¶
- Organisciak, P., & Acar, S. (2026). Know when to trust: Making AI scoring more reliable for educational assessment. Behavior Research. https://doi.org/10.3758/s13428-026-03058-1
Page metadata
| Field | Value |
|---|---|
type |
product |
id |
ocsai-and-confidence-python-libraries-for-confidence-aware-llm-scoring |
title |
Ocsai and confidence Python libraries for confidence-aware LLM scoring |
description |
Python libraries released by Organisciak et al. |
status |
review |
product_kind |
software |
generated |
by process:settle-candidates on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.