GPT-4o's correctness-labeling errors concentrate on final turns requiring numerical calculation, and its main KC-labeling error is assigning too few standards to a turn.¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q1
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q1 i? Most GPT-4o correctness errors are on the final turn, especially where numerical calculations are needed to verify correctness. → Alexander Scarlatos 2024 q1 i? GPT-4o's primary KC-labeling error is not assigning sufficient standards to a turn to describe all required KCs. → Alexander Scarlatos 2024 (2)

Evidence¶

Alexander Scarlatos 2024¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490 doi:10.48550/arxiv.2409.16490

q1 · i? · design · r2

Qualitative error analysis of GPT-4o's annotations: "most of GPT-4o’s errors are on the final turn", for example when a student gives the numerical result of a sum or product; the authors suggest external tools such as calculators may help.

"When labeling correctness, most of GPT-4o’s errors are on the final turn, especially in questions that require some numerical calculations to verify correctness, such as when the student provides the numerical result of a sum or product calculation."

Alexander Scarlatos 2024 (2)¶

Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. (2024). Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs. Published in LAK25: The 15th International Learning Analytics and Knowledge Conference. https://arxiv.org/abs/2409.16490

q1 · i? · design · r2

Qualitative error analysis of KC labels: the "primary error is not assigning sufficient standards to a turn", which the authors trace mostly to not selecting enough relevant domains and clusters early in the recursive annotation.

"When labeling KCs, GPT-4o’s primary error is not assigning sufficient standards to a turn to describe all the required KCs."

Discussion¶

Related Claims¶

• Expert Teachers Rate Gpt 4O Dialogue Annotations As Largely Accurate
• GPT-4o's final-turn correctness labeling on CoMTA is only slightly less accurate than expert human annotators, which the authors read as close to human-level performance on a challenging task. — related
• Adding regex-based rules to few-shot LLM labeling improves agreement with human annotation (0.818 to 0.852 overall) — related
• GPT-4o-mini natural language feedback was overwhelmingly helpful and never harmful in quality annotation — related
• Multimodal prompts caused o4-mini to skip numerical integration and produce only qualitative final responses despite recovering much of the analytical structure — related
• Only the well-specified text-only prompt produced a correct quantitative result from o4-mini, while underspecified prompts led to numerical errors and misleading final results — related
• Some VLMs perform near chance on binary judgments of student correctness, and error-assessment behavior is idiosyncratic across models — related