Adding tutor interface and skill-estimate context to the prompt substantially improves GPT-4's error-diagnosis accuracy¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? On the complex factoring tutors, progressively adding interface information and skill estimates to the prompt raised GPT-4's diagnosis accuracy across error types. → Reddig 2025

Evidence¶

Reddig 2025¶

Reddig, J. M., Arora, A., & MacLellan, C. J. (2025). Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-025-00505-6

q2 · i? · design · r3

Three rounds of prompt engineering on the factoring dataset (N = 199): with basic context GPT-4 questioned whether a correct answer was correct 21% of the time; adding the interface brought correct-answer recognition to 100%, and adding the skill estimate improved logical-mistake diagnosis to 81%.

"Table 3 demonstrates how, with additional context to situate the student's answer within the appropriate sub-problem, GPT-4's accuracy increased on a complex tutoring environment like factoring."

Discussion¶

Related Claims¶

• Given sufficient prompt context, GPT-4 describes student errors in a College Algebra ITS with 87.8% accuracy — a narrower finding that bears on this claim
• GPT-4's error-diagnosis accuracy varies by error type: near-perfect on correct answers, weakest on Wrong Field and multiple-error responses — related
• GPT-4 generates targeted corrective feedback for 66% of student errors, but 35% of hints are too general, incorrect, or give away the answer — related