ITS evaluation results depend strongly on test type: effects are large on locally developed tests (ES = 0.73) and small on standardized tests (ES = 0.13)¶

Claim · All claims
Evidence · 1 study (3 entries) · quant-synthesis r2–r3 · q2–q4
Currency · newest study 2016 · checked against its sources 2026-10-20 (automated, 3 of 3 entries) · not reviewed by a person

Subclaims¶

q4 i? Test type was the study feature most strongly related to effect size: average ES was 0.73 for local tests, 0.13 for standardized tests, and 0.45 for both. → Kulik 2016 q3 i? Homogeneity analyses confirmed the pattern: average g was 0.62 on local tests, 0.09 on standardized tests, and 0.46 on both. → Kulik 2016 (2) q3 i? Within a single study of Practical Algebra Tutor, effects were larger on aligned local tests (mean ES = 0.99) than on standardized tests (mean ES = 0.36). → Kulik 2016 (3)

Evidence¶

Kulik 2016¶

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research. https://doi.org/10.3102/0034654315581420

q4 · i? · quant-synthesis · r3

Univariate ANOVAs on the Winsorized data set (Glass's ES, study as unit) identified test type as the strongest moderator of ITS effects across the 50 evaluations. Local, standardized, and combined test categories averaged 0.73, 0.13, and 0.45 respectively.

"ESs are large in evaluations that used local tests as outcome measures (average ES = 0.73), small in evaluations that used standardized tests (average ES = 0.13), and intermediate in evaluations that used a combination of the two (average ES = 0.45)."

Kulik 2016 (2)¶

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research. https://doi.org/10.3102/0034654315581420

q4 · i? · quant-synthesis · r3

Supplementary Hedges's-g homogeneity analysis of the Winsorized data set confirmed test type as the strongest moderator, with average g of "0.62 when outcomes were measured on local tests, 0.09 when they were measured on standardized tests, and 0.46 when they were measured on both."

"average g was 0.62 when outcomes were measured on local tests, 0.09 when they were measured on standardized tests, and 0.46 when they were measured on both."

Kulik 2016 (3)¶

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research. https://doi.org/10.3102/0034654315581420

q2 · i? · quant-synthesis · r2

The review attributes this within-study pattern to Koedinger et al. (1997), who evaluated the Practical Algebra Tutor on locally developed problem-solving tests and standardized multiple-choice tests; the review reports, citing that study, that local-test effects were larger. The review reports it second-hand (q 2).

"The researchers found large effects on the locally developed tests (mean ES = 0.99) and smaller effects on the standardized ones (mean ES = 0.36)."

Studies bearing on this claim¶

Does the kind of outcome measure (a researcher-made or aligned test against a standardized one) change the size of the effect a study reports? This table gathers the studies behind this claim and the 4 narrower claims that bear on it: 5 distinct studies (3 quant-synthesis, 2 review), q2–q4, 1 of 5 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• The DARPA Digital Tutor, an ITS for U.S. Navy information systems technicians, produced the largest effects in the data set (average ES of 1.97 and 3.18 in two summative evaluations) — related
• The type of CT measure employed (standardized vs. non-standardized; item format) influences evaluation of intervention effectiveness — a narrower finding that bears on this claim
• Language and early writing interventions showed larger effects on researcher-developed than standardized outcome measures, while phonological awareness, print knowledge, and decoding effects were comparable across measure types — related
• Mastery learning effects are larger with a higher mastery standard, group-based pacing, local tests, and less feedback for the comparison group — related