Realism issues were detected by agents during generation but were rare in final versions¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? Teachers made 10 prompting moves explicitly flagging realism issues and 20 moves modifying quantity or unit, while only 6 student comments flagged realism issues in final problems. → Walkington 2025

Evidence¶

Walkington 2025¶

Walkington, C., Beauchamp, T., Ikram, F., Koçyiğit Gürbüz, M., Xia, F., Lee, M., Lan, A. (2025). Mathematics Teachers’ Interactions with a Multi-Agent System for Personalized Problem Generation. https://arxiv.org/abs/2604.12066. doi:10.1007/978-3-032-29770-9_5

q2 · i? · design · r2

Thematic analysis of teacher prompts and student feedback found that although the agents detected many issues with realism as problems were being written, there were few realism issues noted by teachers and students in the final versions. Teachers also made 20 prompting moves modifying quantity or unit, and 6 student comments flagged realism issues.

"There were 10 instances coded under the “realism” theme, where the teacher explicitly let the LLM know that the problem it had created was not realistic"

Discussion¶

Related Claims¶

• Authenticity and realism are the most frequent failure modes in initial LLM-personalized math problems, and a single refinement iteration substantially reduces failures — related
• Validator agent reliability is highest for realism and lowest for authenticity in human evaluation — related
• Validator failure patterns vary by interest topic and math curriculum unit — related
• Mathematical hallucinations were rare but teachers used professional knowledge to adjust mathematical properties — related
• Teachers adjusted problem topics as the most common refinement move — related