LLMKT: LLM-Based Knowledge Tracing for Dialogues

Element · All elements
Evidence · 5 claims (4 for, 1 mixed) · 3 studies (2 design, 1 causal), q1–q2 · 0 of 3 report an effect size · 5 claims rest on one study

Description

LLMKT is the article's knowledge tracing method: "a novel LLM-based KT method, LLMKT, that leverages the textual content in dialogues, by fine-tuning the open-source Llama 3 LLM" on the KT objective. Given the dialogue up to the target turn and a prompt about one KC, it estimates mastery from the logits of the True and False tokens, averages KC masteries into a correctness prediction, and is trained with binary cross entropy. The article reports that "averaging over KC masteries performed better than taking a product over them", and the authors publicly release their code.

Design Implications

Context

Requirements

• Fine-tuning Llama-3.1-8B-Instruct (the article uses LoRA on NVIDIA RTX A6000 GPUs) on turn-level correctness labels.
• The dialogue text up to the current tutor turn and the text of each KC, packed into a single prompt with customized attention masks so KCs do not attend to each other.

Constraints

• KCs and correctness labels for previous turns are not explicitly provided due to memory constraints.
• The methods need to be tested on larger-scale data, since tutoring dialogue datasets are smaller than those typically used in standard KT works.

Target Learners

• Students in one-on-one math tutoring dialogues with human or LLM-powered tutors (the article uses the CoMTA and MathDial datasets)

Target Learning Goals

• Estimating student knowledge of math knowledge components (Common Core standards) and predicting student response correctness across dialogue turns

Affordances

• Dialogue Knowledge Tracing Framework

Claims

• LLMKT outperforms existing knowledge tracing methods at predicting student turn correctness in the CoMTA and MathDial tutoring dialogue datasets, and generally outperforms DKT-Sem. [+W]
• In a qualitative case study, LLMKT adjusts KC mastery estimates using the dialogue's textual content, such as the difficulty of the tutor's question, rather than only prior correctness labels. [+W]
• LLMKT's predicted knowledge change curves on CoMTA are mixed across the 15 most frequent KCs, though overall they mostly resemble the power law of practice when dialogues have sufficient turns. [+W]
• In the fine-tuning experiments, increasing base model size up to 32B parameters improved performance, and larger LLMs consistently outperformed smaller MLMs [~W]
• Applied to the full dataset of 936 students, the LLM reproduced the human-observed growth trends, detecting significant growth on Q25 and Q26 but none on Q27 [+W]

The evidence by question

• Can an LLM-based knowledge tracing model estimate what a student knows from a tutoring dialogue more accurately than standard knowledge tracing, and how accurate is it in absolute terms? LLMKT outperforms existing knowledge tracing methods at predicting student turn correctness in the CoMTA and MathDial tutoring dialogue datasets, and generally outperforms DKT-Sem. [+W]

Related Elements

• Adaptive Learning

Examples

-

Key Sources

• Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490

Page metadata

Field	Value
type	element
id	llmkt-llm-based-knowledge-tracing
title	LLMKT: LLM-Based Knowledge Tracing for Dialogues
description	LLMKT is the article's knowledge tracing method: "a novel LLM-based KT method, LLMKT, that leverages the textual content in dialogues, by fine-tuning the open-source Llama 3 LLM" on the KT objective.
status	review
canonical	True
generated	by claude/unspecified on 2026-09-24
trust	unverified — no one has confirmed this page's content
sources	1 — alexander-scarlatos-2024

Frontmatter is the page's machine-readable record — see the Schema & Guide.