Intelligent tutoring raised test scores by a median of 0.66 standard deviations over conventional instruction across 50 controlled evaluations¶

Claim · All claims
Evidence · 1 study (2 entries) · quant-synthesis r3 · q4 · i2 medium
Currency · newest study 2016 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q4 i2 In the meta-analysis of 50 evaluations, intelligent tutoring raised test scores a median of 0.66 standard deviations (50th to 75th percentile) over conventional instruction. → Kulik 2016 q4 i2 Under a random-effects weighted analysis, the average ITS effect was g = 0.50, 95% CI [0.40, 0.59]. → Kulik 2016 (2)

Evidence¶

Kulik 2016¶

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research. https://doi.org/10.3102/0034654315581420

q4 · i2 · quant-synthesis · r3

Meta-analysis of 50 controlled evaluations of intelligent tutoring systems, using Glass's ES, study as unit, unweighted means, with a 90% Winsorized set for outliers. The article prints the effect size median ES = 0.66, unchanged after Winsorizing, where "The median ES in the original data set is 0.66."

"The median ES in the original data set is 0.66. The mean ES is 0.65; the standard deviation is 0.56. In the Winsorized data set, median is 0.66, mean is 0.61, and standard deviation is 0.38."

Kulik 2016 (2)¶

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research. https://doi.org/10.3102/0034654315581420

q4 · i2 · quant-synthesis · r3

Supplementary random-effects weighted analysis with evaluation report as unit, using Hedges's g. The printed effect size was g = 0.50: "the average g = 0.50, 95% CI [0.40, 0.59], p < .001", slightly below the unweighted estimate.

"With evaluation report as the unit of analysis and weighting based on a random-effects model, the average g = 0.50, 95% CI [0.40, 0.59], p < .001."

Discussion¶

Related Claims¶

• The DARPA Digital Tutor, an ITS for U.S. Navy information systems technicians, produced the largest effects in the data set (average ES of 1.97 and 3.18 in two summative evaluations) — a narrower finding that bears on this claim
• Intelligent tutoring systems improve achievement over non-individualised instruction — related
• ITS groups outperformed conventional-instruction controls in 92% of the 50 evaluations, with gains above 0.25 SD in 78% — a narrower finding that bears on this claim