Human-tutoring transcript analysis informed BEETLE II tactics, including defaulting to the full answer after three unsuccessful elicitation attempts¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person
Design limits · Dzikovska 2014 (2): no comparison group (read from the study by two models, 2026-10-27)

Subclaims¶

q2 i? Analysis of human tutoring dialogues showed that mini-tutorials unsuccessful after three tutor turns were rarely ever successful, so the system defaults to telling the student the answer after three failed elicitation attempts. → Dzikovska 2014 q2 i? A hierarchical coding scheme with eight lowest-level tactic codes was developed to analyze tutor techniques; three raters achieved kappa scores of 0.69 to 0.88 across coding levels. → Dzikovska 2014 (2)

Evidence¶

Dzikovska 2014¶

Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., & Campbell, G. (2014). BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics. International Journal of Artificial Intelligence in Education, 24, 284–332. https://doi.org/10.1007/s40593-014-0017-9

q2 · i? · design · r2

Pilot study in which three experienced human tutors taught the curriculum to students via chat; thirty students participated, each for approximately four hours. Transcripts of mini-tutorial dialogues were classified by tactic sequence and success, and the most frequent successful sequences became system tactics. No effect size is printed.

"Our analysis also indicated that tutorial dialogues that were not successful after three tutor turns were rarely e ver successful. Hence, we recommended that the BEETLE II system default to providing the full and correct answer to the student if three attempts at eliciting the answer were not successful."

Dzikovska 2014 (2)¶

Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., & Campbell, G. (2014). BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics. International Journal of Artificial Intelligence in Education, 24, 284–332. https://doi.org/10.1007/s40593-014-0017-9

q2 · i? · causal · r2

Reliability analysis of the pilot's hierarchical coding scheme for low-level tutor techniques, which used eight tactic codes at the lowest level. Inter-rater agreement ranged from moderate to substantial (kappa 0.69 to 0.88). No effect size is printed.

"Three independent raters achieved kappa scores ranging from 0.69 to 0.88 across the levels of the coding hierarchy, indicating agreement ranging from moderate to substantial."

Discussion¶

Related Claims¶

• Dynamically generated natural-language feedback was not demonstrated to be superior to a non-NLP feedback condition that told students the answer — related
• The BEETLE II curriculum produces large pre- to post-test learning gains relative to a no-training control group (effect size 1.72) — related
• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — related