LLM misconception analysis that ties student errors to instructional content reveals insights not apparent from performance data alone¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? GPT-4 analysis combining quiz questions, most common incorrect answers, and lecture transcripts produces misconception hypotheses linked to how material was taught. → Parker 2025

Evidence¶

Parker 2025¶

Parker, M. J., & Zavala-Cerna, M. G. (2025). What Don't You Understand? Using Large Language Models to Identify and Characterize Student Misconceptions About Challenging Topics. https://arxiv.org/abs/2605.00294 doi:10.48550/arxiv.2605.00294

q2 · i? · design · r2

Qualitative demonstration using an Immunology checkpoint blockade example and a Physiology acid-base example. "This relationship between student answers and instructional material would not be apparent without providing the model with the content as context", showing misconceptions tied to specific instructional framing.

"This relationship between student answers and instructional material would not be apparent without providing the model with the content as context"

Discussion¶

Related Claims¶

• LLM-generated misconceptions grounded in quiz content, response patterns, and lecture transcripts are rated excellent by subject matter experts — related
• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — related
• A hybrid human-AI workflow using GPT-4o with retrieval-augmented generation supported efficient inductive thematic analysis while preserving researcher judgment — related
• LLM-generated ideas diverge from lesson ideas in grounding, misconception substance, vocabulary, uncertainty markers, and analogy use — related
• Frontier LLMs achieve the best balance between detecting misconceptions in correct-answer cases and not over-flagging correct students, while fine-tuned local models achieve under 58% TM recall — related
• GPT-4's error-diagnosis accuracy varies by error type: near-perfect on correct answers, weakest on Wrong Field and multiple-error responses — related