Adopt unified KPIs and rigorously assess learner diversity, recommender fairness, and long-term engagement when evaluating ITS¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The review recommends that future ITS research strengthen its empirical foundation by adopting unified key performance indicators and rigorously assessing learner diversity, fairness in recommender systems, and long-term learner engagement. It states that "future research should adopt unified key performance indicators (KPIs), as partially illustrated in Table 3, and rigorously assess learner diversity, fairness in recommender systems, and long term learner engagement". Such efforts, the review argues, would support more consistent benchmarking and promote scalable and equitable deployment of ITS in real-world educational environments.

Design Implications¶

Context¶

Requirements¶

• Evaluation methodologies must move beyond the current inconsistency between subjective engagement indicators and standardized performance assessments

Constraints¶

• The review notes widespread adoption remains constrained by unresolved challenges including student data privacy, algorithmic bias, and academic integrity

Target Learners¶

• Diverse learner profiles across educational settings, including low-resource language and underrepresented learner populations

Target Learning Goals¶

• Consistent benchmarking of ITS pedagogical effectiveness and user satisfaction
• Long-term learner engagement and fairness in personalized recommendations

Related Strategies¶

• Future AI-SRL research should broaden search strategies, include educators and younger learners, address motivation, and apply longitudinal designs
• Adopt more precise terminology such as digital personalized learning (DPL) or personalized adaptive learning, and pursue rigorous longitudinal research with younger learners
• Adopt standardized reporting protocols for AR enhanced analytics studies and extend analytics into broader XR (VR/MR) contexts
• Diversify AIED: use diverse technologies, address big problems, and reuse existing resources rather than reinventing content

Examples¶

-

Key Sources¶

• Meriem Zerkouk, Miloud Mihoubi and Belkacem Chikhaoui. (2025). A Comprehensive Review of AI-based Intelligent Tutoring Systems: Applications and Challenges. arXiv preprint. https://arxiv.org/abs/2507.18882