One-shot GPT-4o-mini codebook-based annotation of LLM math-tutor dialogue¶
Research Method · All research methods
Evidence · 3 claims (1 for, 2 mixed) · 3 studies (1 causal, 1 associational, 1 design),q2· 1 of 3 report an effect size · 3 claims rest on one study
Description¶
The article reproduces an 82-code educational codebook organised into 7 major categories (MPS, MKT, Cognition, Metacognition, PIU, Student Intent Utterance, Interaction/Discourse) by inlining it into a single GPT-4o-mini prompt at temperature 0: "we reproduce an 82-code educational codebook on119,009tutor sentences with a one-shot structural classifier". Labels cover 3,500 scenarios across Base Greedy, Aligned GRPO, and Base TEI@4 cells, with no human coding.
Accounts¶
- One-shot GPT-4o-mini reproduction of an 82-code educational codebook on 119,009 tutor sentences: The article reproduces an 82-code educational codebook organised into 7 major categories (MPS, MKT, Cognition, Metacognition, PIU, Student Intent Utterance, Interaction/Discourse) by inlining it into a single GPT-4o-mini prompt at temperature 0: "we reproduce an 82-code educational codebook on119,009tutor sentences with a one-shot structural classifier". Labels cover 3,500 scenarios across Base Greedy, Aligned GRPO, and Base TEI@4 cells, with no human coding. (The Tutoring Effectiveness Index: Predicting LLM Math Tutor Quality from Four Conversation)
Claims¶
- GRPO pedagogical alignment of a reasoning LLM lowers actual student learning while raising judge-rated helpfulness (a multi-dimensional alignment tax) [+M]
- On codes requiring recognition of pedagogical intent not explicitly stated, LLMs over-apply and the blind verifier rejects their application in favor of human coding [~W]
- Cosine similarity has opposing effects on LLM coding accuracy: lower similarity within codebooks helps, higher similarity within excerpts helps [~W]
Related Research Methods¶
-
Key Sources¶
- Shim Jaechang and Unggi Lee. 2026. The Tutoring Effectiveness Index: Predicting LLM Math Tutor Quality from Four Conversation Signals. In Proceedings of The 35th ACM International Conference on Information and Knowledge Management (CIKM '26). https://arxiv.org/abs/2605.30666
Page metadata
| Field | Value |
|---|---|
type |
research-method |
id |
one-shot-gpt-4o-mini-codebook-based-annotation-of-llm-math-tutor-dialogue |
title |
One-shot GPT-4o-mini codebook-based annotation of LLM math-tutor dialogue |
description |
The article reproduces an 82-code educational codebook organised into 7 major categories (MPS, MKT, Cognition, Metacognition, PIU, Student Intent Utterance, Interaction/Discourse) by inlining it into a single GPT-4o-mini prompt at temperature 0: "we reproduce an 82-code educational codebook on119,00 |
status |
review |
generated |
by process:settle-candidates on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.