Evaluate LLM-based tutors with human-inspired methods such as simulated students and behavior analysis before classroom deployment¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The authors recommend that AIED researchers develop systematic, human-inspired evaluation techniques for learner-facing LLM tools before practical use, since experimental studies of learning outcomes are "usually costly, time-consuming, and inefficient". Proposed approaches include behavior analysis of how closely LLMs address stakeholders' downstream needs, AI teacher tests based on human evaluation of dialogue, and using student models and simulated students to role-play learners' processes—for example probing whether an LLM math tutor addresses a specific misconception or keeps asking questions when a student gets stuck.

Design Implications¶

Context¶

Requirements¶

• Systematic approaches to evaluate the affordances and limitations of current LLMs in education, which the authors say are currently lacking
• Attention to the methodological and ethical limitations of using LLMs, especially black box proprietary models, in education and research

Constraints¶

• Requiring human evaluation of dialogue can be costly and may not cover behaviors that show up infrequently or only after many turns of dialog with a real student
• Evaluating AI models in multi-turn dialogues remains an open question in AI research

Target Learners¶

• AIED researchers and developers of learner-facing AI tutoring tools

Target Learning Goals¶

• Rigorous evaluation of AI tutors' pedagogical behavior before deployment with learners

Related Strategies¶

• Use reasoning-aware student simulators to evaluate AI tutors, diagnose misconceptions, and support learner reflection
• Simulation-based pre-validation of adaptive coaching systems before classroom deployment
• Use LLM persona simulation only for preliminary experimentation, stress testing, and hypothesis generation, with separate validation for consequential applications
• Test whether success depends on compiled knowledge via knowledge ablation and downstream Knowledge Tracing validation

Examples¶

-

Key Sources¶

• Sina Rismanchian & Shayan Doroudi. (2025). The Evolution of Research on AI and Education Across Four Decades: Insights from the AIxEd Framework. Int J Artif Intell Educ 35:2797–2820. https://doi.org/10.1007/s40593-025-00483-9