Using a WATS in community college elementary algebra corresponds to a statistically significant 2.59-point post-test gain (Hedges g = 0.32) controlling for pretest¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3 · i1 small
Currency · newest study 2017 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Controlling for students' pretest scores, WATS use corresponded to a 2.59 point increase in end-of-semester MDTP post-test scores, statistically significant at p < .05, equivalent to about 5 percentage points on a 50-point test. → Shandy Hauk 2017 q3 i1 The effect's Hedges g is 0.32 with a 95% confidence interval of .14-.50, which the authors call a small but noteworthy effect for studies of this size. → Shandy Hauk 2017 (2)

Evidence¶

Shandy Hauk 2017¶

Shandy Hauk, Bryan Matlen, Larry Thomas. (2017). Exploration of the Factors that Support Learning: Web-based Activity and Testing Systems in Community College Algebra. 20th Annual Conference on Research in Undergraduate Mathematics Education. https://sigmaa.maa.org/rume

q3 · i? · causal · r2

Hierarchical linear modeling of MDTP post-test scores from 510 students of 29 instructors across 18 colleges in a multi-site cluster randomized trial, with random teacher effects and student- and teacher-level pretest covariates. The WATS coefficient was 2.59 (SE 1.17, p = .04); the article reports "a statistically significant positive effect (p < .05)" but no effect size in this model result.

"Controlling for students' pretest scores, we found that using WATS corresponded to a 2.59 point increase in students' post-test scores, a statistically significant positive effect (p < .05)."

Shandy Hauk 2017 (2)¶

Shandy Hauk, Bryan Matlen, Larry Thomas. (2017). Exploration of the Factors that Support Learning: Web-based Activity and Testing Systems in Community College Algebra. 20th Annual Conference on Research in Undergraduate Mathematics Education. https://sigmaa.maa.org/rume

q3 · i1 · causal · r2

Standardized effect size reported for the same HLM analysis of the cluster randomized trial's first cohort: "The Hedges g value for this effect is 0.32", with a 95% confidence interval of .14 - .50, characterized by the authors as small but noteworthy.

"The Hedges g value for this effect is 0.32, which is considered a small but noteworthy effect in educational research for studies of this size (Cheung & Slavin, 2015; Hill et al. 2008). The 95% confidence interval of the Hedges g value is .14 - .50."

Discussion¶

Related Claims¶

• Cognitive apprenticeship instruction produced no significant differences from traditional instruction on post-treatment problem-solving, final, and standardized mathematics exams in community college technical mathematics — related
• ChatGPT 3.5 produced low-quality hints for 32% of problems, reducible to near 0% for algebra and 13% for statistics via the self-consistency technique — related
• ASSISTments use in seventh grade produces a statistically significant long-term math achievement effect of 0.10 SD one year after implementation ends — related
• ASSISTments use in grade 7 math homework produces a statistically significant long-term positive effect on grade 8 state test scores one year after the intervention ends — related
• The WATS efficacy estimate is qualified by high instructor attrition, and replication with a second cohort is underway — related
• Adaptive web-based system treatment showed a small, statistically non-significant advantage over control on elementary algebra post-test scores (Hedges' g = .12) — reports the opposite
• Treatment but not control instructors reported an increase in their ability to use technology for teaching mathematics after a practice semester, though not statistically significant — related