Apply error mitigation such as self-consistency before deploying LLM-generated help, and frame unmitigated LLM feedback as an imperfect source¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends that educators and system designers not integrate raw LLM output into pedagogy without error mitigation or awareness of domain-specific error rates. Self-consistency — generating multiple samples and keeping the modal answer — reduced hint error to near 0% in algebra. Where error cannot be reduced to near zero, the authors suggest "framing ChatGPT-produced feedback as coming from an “imperfect robot” or peer-like source of information so that students may consider its responses critically."

Design Implications¶

Context¶

Requirements¶

• A quality-check procedure (correct answer, correct work, no inappropriate language) applied to generated hints before learner use

Constraints¶

• Self-consistency left a 13% error rate for statistics, so near-zero error was achieved only in algebra domains

Target Learners¶

• secondary and early post-secondary mathematics learners

Target Learning Goals¶

• receiving accurate worked-solution hints during problem solving

Related Strategies¶

• Evaluate AI systems before deploying them with students
• Evaluate and mitigate sentiment bias across sensitive attributes before deploying LLM forum support
• Encourage students to use AI as a conversational debugging guide while informing them of LLM imperfections and cautioning against unjustified confident claims
• Combine data-driven and expert-driven approaches, and add a filter network, to mitigate non-factual output in automated feedback
• Design teachable ChatGPT sessions with purposeful errors or lower-level models so learners practice error correction
• Benchmark AI assistants on specialized educational tasks before classroom deployment
• Introduce prompt-engineering training before students use Gen AI for structured self-study techniques
• Build infrastructure-level mechanisms to monitor, benchmark, and control LLM response behavior across model transitions in conversation-based assessment
• Mitigate LLM feedback errors by supplying course materials and compiler errors in the prompt, fact-checking generated feedback, and adapting feedback structure to student behavior

Examples¶

-

Key Sources¶

• Pardos ZA, Bhandari S (2024) ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLoS ONE 19(5): e0304013. https://doi.org/10.1371/journal.pone.0304013