LLMs predict common distractors above chance but fall 15 percentage points below chance at predicting which errors students rarely select¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Models achieve 47.9% accuracy identifying which wrong answer students choose most often, exceeding the 35.8% random baseline, but fall 15 percentage points below chance predicting least-selected distractors. → Eamon Worden 2026
Evidence¶
Eamon Worden 2026¶
Eamon Worden, Cristina Heffernan, Neil Heffernan, and Shashank Sonkar. (2026). FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs. https://arxiv.org/abs/2602.00070 doi:10.48550/arxiv.2602.00070
q2 · i? · design · r2
Zero-shot evaluation on most- and least-common distractor prediction tasks over 236 multiple-choice problems with at least 10 responses. The article reports the asymmetry quoted above and concludes LLMs "have learned to recognize some common misconceptions but cannot distinguish likely from unlikely distractors".
"models achieve 47.9% accuracy at identifying which wrong answer students choose most often, exceeding the 35.8% random baseline, but fall 15 percentage pointsbelowchance when predicting which errors students rarely select."
Discussion¶
Related Claims¶
- AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
- Cognitive modeling of exact student responses is hardest for multi-select problems and rare-error answers, with select-all accuracy of only 12-15% and cognitive accuracy dropping to as low as 3.7% when answers are incorrect — related
- LLMs perform competently at comparing item difficulty but fall below random chance on comparing item discrimination — related
- LLM error simulation rarely fails, but the model is less likely to reproduce a targeted distractor on problems with longer correct solutions — related
- On the misconception-annotated math dataset, LLMs predominantly follow a solve-first, error-injection process aligned with misconception-based distractor design — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-distractor-prediction-asymmetry |
title |
LLMs predict common distractors above chance but fall 15 percentage points below chance at predicting which errors students rarely select |
description |
LLMs predict common distractors above chance but fall 15 percentage points below chance at predicting which errors students rarely select |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — eamon-worden-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.