Knowledge tracing performance on tutoring dialogues is relatively low, with a maximum of around 76% AUC, which the authors take to show dialogueKT is a challenging task.¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q1–q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? Performance of all KT methods is relatively low on dialogueKT, with a maximum of around 76% AUC for student turn correctness prediction. → Alexander Scarlatos 2024 q1 i? The authors attribute the difficulty partly to student behavior in dialogues being more unpredictable than in traditional KT, given many diverse types of student responses. → Alexander Scarlatos 2024 (2)

Evidence¶

Alexander Scarlatos 2024¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490 doi:10.48550/arxiv.2409.16490

q2 · i? · design · r2

Benchmark comparison across both datasets: performance is "relatively low on the dialogueKT task, with a maximum of around 76% AUC", which the article contrasts with standard KT methods surpassing 80% AUC on some datasets.

"Finally, we see that performance of all KT methods is relatively low on the dialogueKT task, with a maximum of around 76% AUC for student dialogue turn correctness prediction."

Alexander Scarlatos 2024 (2)¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490

q1 · i? · design · r2

Authors' explanation of the low performance, not a tested result: "student behavior in dialogues is more unpredictable compared to traditional KT". A first stated reason is that CoMTA dialogues are short, with only 4-5 labeled turn pairs on average.

"Second, student behavior in dialogues is more unpredictable compared to traditional KT, since there can be many diverse types of student responses to tutor turns."

Discussion¶

Related Claims¶

• Llmkt Outperforms Existing Kt Methods On Tutoring Dialogues
• Existing KT methods fail to beat a majority-class baseline on the small CoMTA dialogue dataset but perform significantly better on the larger MathDial dataset. — related
• DKT-Sem, a DKT variant using semantic text embeddings, performs better than existing KT methods on tutoring dialogues, with a smaller margin on the larger MathDial dataset. — related
• GPT-4o's final-turn correctness labeling on CoMTA is only slightly less accurate than expert human annotators, which the authors read as close to human-level performance on a challenging task. — related
• LLMKT's predicted knowledge change curves on CoMTA are mixed across the 15 most frequent KCs, though overall they mostly resemble the power law of practice when dialogues have sufficient turns. — related
• Standardized benchmarks are often too generic to reflect the needs of innovative educational applications — related
• In system cold-start scenarios, CLST outperformed every baseline KT model across all five datasets, with the largest gains at 8 training students — related
• A pedagogically informed Tutor Prompt yields higher Success@N and lower Telling@N than MathDial's Base Prompt in simulated tutor-student dialogues — related
• Among deep learning KT models, simpler DKT performs comparably to complex architectures under limited dialogue data — related
• An IRT-based difficulty-aware conversational KT framework improves AUC over the LLMKT baseline on both QATD2k and MathDial — related
• Extended multi-turn dialogue does not produce effective tutoring: learning gains remain uniformly low and misconception reduction peaks around 24-25% — related
• Learner attributes differ sharply in how reliably they can be recovered from tutoring dialogue, with behavioral traits slow to surface — related
• Hybrid approaches combining LLMs with traditional knowledge-tracing algorithms outperform either method alone for learner modelling
• Frontier LLMs barely exceed a trivial baseline on knowledge tracing, with the best model surpassing always-predict-correct by only about five percentage points — related