LLMKT's predicted knowledge change curves on CoMTA are mixed across the 15 most frequent KCs, though overall they mostly resemble the power law of practice when dialogues have sufficient turns.¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q1
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q1 i? Of the 15 most frequent KCs, 5 trend towards increasing mastery, 5 towards decreasing mastery, and 5 do not change significantly. → Alexander Scarlatos 2024 q1 i? Accounting for the small data and truncated dialogues, the curves seem to mostly resemble the power law of practice, with predicted mastery increasing before leveling out when there are sufficient turns. → Alexander Scarlatos 2024 (2)

Evidence¶

Alexander Scarlatos 2024¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490 doi:10.48550/arxiv.2409.16490

q1 · i? · design · r2

Learning-curve analysis of LLMKT's predicted masteries on CoMTA test splits, for the 15 most frequent KCs: "the results are mixed: 5 out of 15 KCs trend towards increasing mastery", 5 toward decreasing mastery and 5 unchanged. No statistics are printed.

"Out of these, the results are mixed: 5 out of 15 KCs trend towards increasing mastery as the dialogues go on, 5 KCs trend towards decreasing mastery, and 5 KCs do not change significantly in mastery."

Alexander Scarlatos 2024 (2)¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490

q1 · i? · design · r2

Authors' overall reading of the same learning-curve analysis: the trends "seem to mostly resemble the power law of practice" once the small amount of data and truncated CoMTA dialogues are taken into account.

"Overall, when we take into account the small amount of data and how dialogues in the CoMTA dataset are truncated, the knowledge change curve trends seem to mostly resemble the power law of practice; when there are sufficient turns in a dialogue, the student’s predicted mastery increases before leveling out eventually."

Discussion¶

Related Claims¶

• Knowledge tracing performance on tutoring dialogues is relatively low, with a maximum of around 76% AUC, which the authors take to show dialogueKT is a challenging task. — related
• In a qualitative case study, LLMKT adjusts KC mastery estimates using the dialogue's textual content, such as the difficulty of the tutor's question, rather than only prior correctness labels. — related
• LLMKT outperforms existing knowledge tracing methods at predicting student turn correctness in the CoMTA and MathDial tutoring dialogue datasets, and generally outperforms DKT-Sem. — related
• Providing the tutor with student mastery and practice-history context improved engagement and next-item correctness — related
• Separating ability estimation from correctness prediction yields smoother, more stable learning trajectories than probability-based mastery estimation — related