LLMKT outperforms existing knowledge tracing methods at predicting student turn correctness in the CoMTA and MathDial tutoring dialogue datasets, and generally outperforms DKT-Sem.¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2024 · 1 of 2 entries checked against their sources (automated) · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? LLMKT significantly outperforms all existing KT methods (BKT, DKT, DKVMN, AKT, SAINT, simpleKT) and generally outperforms DKT-Sem on both datasets. → Alexander Scarlatos 2024 q2 i? On the larger MathDial dataset, LLMKT still maintains a large advantage over all other KT methods. → Alexander Scarlatos 2024 (2)

Evidence¶

Alexander Scarlatos 2024¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490 doi:10.48550/arxiv.2409.16490

q2 · i? · design · r2

Benchmark comparison of KT methods on two math tutoring dialogue datasets, scored by accuracy, AUC and F1 (Table 2). The authors report that "LLMKT significantly outperforms all existing KT methods and generally outperforms DKT-Sem on both datasets"; no effect size is printed.

"We see that LLMKT significantly outperforms all existing KT methods and generally outperforms DKT-Sem on both datasets, confirming its effectiveness in the dialogue setting."

Alexander Scarlatos 2024 (2)¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490

q2 · i? · design · r2

MathDial results from the same benchmark comparison, where existing methods improve with more data: "LLMKT still maintains a large advantage over all other KT methods". No test statistic or effect size is printed in this sentence.

"However, LLMKT still maintains a large advantage over all other KT methods."

Studies bearing on this claim¶

Can an LLM-based knowledge tracing model estimate what a student knows from a tutoring dialogue more accurately than standard knowledge tracing, and how accurate is it in absolute terms? This table gathers the studies behind this claim and the 10 narrower claims that bear on it: 4 distinct studies (3 design, 1 theoretical), q1–q2, 0 of 4 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Dkt Sem Outperforms Existing Kt Methods Most With Little Training Data
• Dialogue Kt Performance Is Relatively Low Compared To Standard Kt
• Existing KT methods fail to beat a majority-class baseline on the small CoMTA dialogue dataset but perform significantly better on the larger MathDial dataset. — related
• In a qualitative case study, LLMKT adjusts KC mastery estimates using the dialogue's textual content, such as the difficulty of the tutor's question, rather than only prior correctness labels. — related
• LLMKT's predicted knowledge change curves on CoMTA are mixed across the 15 most frequent KCs, though overall they mostly resemble the power law of practice when dialogues have sufficient turns. — related
• Standardized benchmarks are often too generic to reflect the needs of innovative educational applications — related
• Representing exercises by KC name descriptions outperformed ID-based representation when aligning an LLM to knowledge tracing — related
• Among deep learning KT models, simpler DKT performs comparably to complex architectures under limited dialogue data — related
• An IRT-based difficulty-aware conversational KT framework improves AUC over the LLMKT baseline on both QATD2k and MathDial — related
• Deep Knowledge Tracing substantially outperforms Bayesian Knowledge Tracing on knowledge tracing benchmarks — related
• LLMs show a systematic optimism bias in knowledge tracing, predicting far more accurately for students who answer correctly than for those who answer incorrectly — related
• Hybrid approaches combining LLMs with traditional knowledge-tracing algorithms outperform either method alone for learner modelling — related
• Frontier LLMs barely exceed a trivial baseline on knowledge tracing, with the best model surpassing always-predict-correct by only about five percentage points — related