AutoTutor produces learning gains averaging about 0.8 standard deviations over static-textbook controls, and bystanders cannot distinguish its tutoring turns from a human tutor's¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2 · q2 · i3 large
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Across domains, AutoTutor produced learning gains averaging about 0.8 σ above controls who read static instructional materials for equivalent time. → Nye 2014 q2 i? In a bystander Turing test, third-person evaluators of transcripts could not differentiate AutoTutor's turns from a human tutor's. → Nye 2014 (2)

Evidence¶

Nye 2014¶

Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-014-0029-5

q2 · i3 · review · r2

Narrative review synthesis by the authors of AutoTutor evaluation studies across domains. The review reports a printed effect size of 0.8 σ (standard deviation units) above static-material controls; gains are stated to be on par with expert human tutors in computer-mediated conversation.

"AutoTutor has produced learning gains that are on average about 0.8 σ (standard deviation units) above controls who read static instructional materials (e.g., textbooks) for an equivalent amount of time"

Nye 2014 (2)¶

Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-014-0029-5

q2 · i? · review · r2

Review reports the bystander Turing test evaluation in which outsiders judged individual conversational turns from transcripts; no effect size is printed, so impact is null.

"a bystander Turing test was conducted to determine whether outsiders could discriminate if a specific tutoring turn was produced by AutoTutor or a human tutor. Bystanders could not differentiate between AutoTutor and the human tutor"

Discussion¶

Related Claims¶

• AutoTutor's learning gains are virtually equivalent to those of 1-to-1 human tutoring in direct comparisons — related
• AutoTutor achieved learning gains comparable to human tutors in physics and computer literacy — related
• AutoTutor yields average learning gains of 0.3 to 0.8 sigma compared to reading text for an equivalent amount of time — related
• In WHY2 physics studies, AutoTutor outperformed read-textbook and no-materials controls and performed comparably to WHY2/Atlas and non-expert human tutors — related
• Meta-analytic evidence indicates intelligent tutoring systems yield positive learning outcomes compared with non-ITS conditions — a broader claim this one bears on
• Early AutoTutor versions gained about 0.5 σ over textbook controls, with gains depending on test type (cloze 0.64 σ, deep 0.28 σ, shallow 0.15 σ) — related