Conversation-based assessment chat log corpus¶
Product or Programme · All products and programmes
Evidence · 5 claims (4 for, 1 against) · 1 study (1 design),q2· 0 of 1 report an effect size · 5 claims rest on one study
Description¶
A research dataset created by Ober et al. containing more than 10,000 student dialogue turns from English-learning middle school students interacting with a conversation-based mathematics assessment and its pedagogical agents.
Components¶
- Conversation-based assessment (CBA) chat log corpus for middle school mathematics: A chat log dataset of over 10,000 student turns (human coder 1 coded N = 10,919 turns) collected from 107 middle school students classified as English learners (mean age 12.4 years) interacting with a conversation-based assessment for middle school mathematics. The CBA comprised "a sequence of six structured, interactive conversations between the student and three pedagogical agents, a teacher agent and two peer agents," designed to simulate small-group mathematical problem solving with NLP-based dialogue adaptation. Students completed up to seven tasks targeting six competencies including proportional reasoning. (Ober et al. (2026))
Claims¶
- Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs [+W]
- Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none [-W]
- Same-model LLM configuration pairs agree more than cross-model pairs, and agreement decreases monotonically as temperature difference increases [+W]
- LLM configurations show high within-configuration reliability when re-coding the same chat log dataset, with ChatGPT4o/temperature=0 highest [+W]
- Temperature-construct interactions: constructs with moderate theoretical coherence benefited from higher temperatures, while well-defined constructs required deterministic settings [+W]
Related Products and Programmes¶
-
Key Sources¶
- Ober, T. M., Zhang, S., Zapata-Rivera, D., Schroeder, N. L., & Botelho, A. F. (2026). Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent. Journal of Educational Data Mining, Volume 18, No 1. https://doi.org/10.17605/osf.io/s85ck
Page metadata
| Field | Value |
|---|---|
type |
product |
id |
conversation-based-assessment-chat-log-corpus |
title |
Conversation-based assessment chat log corpus |
description |
A research dataset created by Ober et al. |
status |
review |
moved_from |
element/cba-middle-school-math-chat-log-corpus |
product_kind |
dataset |
generated |
by process:sweep-kinds on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — ober-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.