Verbalized self-reported confidence is overconfident in the mid-confidence range, while consistency-based confidence shows good average calibration but localized failure regions¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Verbalizing-based confidence exceeds empirical accuracy in the mid-confidence range (MCE 0.259); consistency-based confidence attains the lowest ECE (0.029) but a high MCE (0.279). → Longwei Cong 2026

Evidence¶

Longwei Cong 2026¶

Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, and Ulf Kroehne. (2026). Confidence Estimation in Automatic Short Answer Grading with LLMs. https://arxiv.org/abs/2605.00200 doi:10.48550/arxiv.2605.00200

q2 · i? · design · r2

Reliability analysis (Fig. 2, Table 1) on the SciEntsBank Test_UD split. Verbalizing-based confidence showed "clear deviations from the diagonal, particularly in the mid-confidence range" with MCE 0.259; consistency-based confidence had Brier 0.186, ECE 0.029, MCE 0.279.

"Consistency-based confidence follows the diagonal more closely and attains the lowest ECE, suggesting good average calibration. However, its relatively high MCE indicates that substantial calibration errors still occur in certain confidence regions"

Discussion¶

Related Claims¶

• The hybrid confidence measure with aleatoric uncertainty yields the best calibration (Brier 0.138, ECE 0.044, MCE 0.100) among compared methods — related
• Latent (token-probability) confidence is the weakest single-source confidence signal for LLM-based ASAG, with the lowest AUROC and worst calibration — related
• A theoretical article, citing two simulation studies, reports that novice clinical learners overestimated their performance and that their confidence exceeded their accuracy — related
• Prior Knowledge Needed For Accurate Self Assessment — related
• Feedback improves confidence calibration, and prior lure exposure inflates confidence in intrusions — related
• A hybrid confidence measure combining model-based signals with dataset-derived aleatoric uncertainty achieves the highest AUROC and strongest accuracy gains in selective grading of LLM-graded short answers — related