Increased solution count correlated with decreased LLM accuracy, falling short of statistical significance except at solution count 5¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? For adjusted accuracy the decrease reached significance only at solution counts of 5 (OR = 3.95, 95% CI [1.11, 18.6], p = 0.048). → Cadiente 2024

Evidence¶

Cadiente 2024¶

Cadiente, A., Chen, J., Kasselman, L., & Pilkington, B. (2024). Large Language Models take on the AAMC Situational Judgment Test: Evaluating Dilemma-Based Scenarios. Research Square preprint. https://doi.org/10.21203/rs.3.rs-4560463/v1

q2 · i? · associational · r2

Ordinal logistic regression of adjusted accuracy on solution count. Table III prints OR = 3.95 (95% CI 1.11, 18.6; p = 0.048) for solution count 5 versus 4; the authors caution the isolated value needs further study.

"Likewise, across all 3 platforms summatively, as solution count increases, the odds of being accurate using adjusted scores also decreases with statistical signi cance only noted in solution counts of 5 (Table II)."

Discussion¶

Related Claims¶

• ChatGPT-4.0 outperformed ChatGPT-3.5 and Bard on the AAMC SJT in both raw and adjusted accuracy — related
• On MedQA, GPT-5.4-mini's accuracy increases monotonically with prompt sophistication across semantically equivalent questions, with generated Low Literacy prompt variants scoring significantly below Prompt Engineer variants — related