LLM-generated retrieval practice was practical to deploy and well received, but high correctness rates (about 70% of items at p=1.0) restricted score variance¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? In-class deployment of LLM-generated MCQs succeeded logistically, with approximately 70% of items achieving a 100% correct response rate. → Yuan An 2026

Evidence¶

Yuan An 2026¶

Yuan An, Lei Wang. (2026). Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses. https://arxiv.org/abs/2609.27063 doi:10.48550/arxiv.2609.27063

q2 · i? · associational · r1

Deployment analysis of 311 in-class MCQs across 52 quizzes (7,888 responses from 54 participants) found "approximately 70% of items achieving a 100% correct response rate (p= 1.0)", limiting variance for item statistics.

"Student scores were high overall, with approximately 70% of items achieving a 100% correct response rate (p= 1.0). This performance level restricts score variance"

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
• LLM-assigned difficulty labels fail to predict empirical item difficulty in classroom retrieval practice (ρ=0.06), as do co-generated Bloom levels (ρ=0.02) — related
• On the live single-node deployment, short-answer rounds take ≈32 s (≈$0.015/round) while multiple-choice rounds are ≈4× faster (≈7.6 s, ≈$0.005/round) because grading is deterministic — related
• The AI feedback system returns responses in under a second for MCQs and about six seconds for open-ended questions, and its reusable prompt is roughly 2.77 times more time-efficient to prepare than equivalent educator text — related
• ConcepTests with roughly 35% to 70% of students answering correctly before discussion appear to produce the highest engagement and most effective discussions — related
• Prebuilt quiz banks of 435 questions across 65 modules decouple practice from live generation, with warm practice-request medians of 4.8 and 5.8 ms — related
• Preservice educators rate generative AI highest for logistical support (time saving, clarity, organizing thought) and lowest for epistemic trust, accuracy and consistency — related
• Students rated ChatGPT highly for completing activities more quickly, among the highest-scoring survey items in both courses — related
• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related