OLI has mixed effects on academic achievement among postsecondary students¶

Claim · All claims
Evidence · 3 studies · 3 causal r2–r3 · q2–q3 · i2 medium
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 One RCT found a statistically significant positive effect of OLI on CAOS statistics scores (effect size 0.55, improvement index +21). → lovett-caos q2 i? One quasi-experimental study found an indeterminate, non-significant negative effect on final exam scores (effect size -0.07). → schunn-final q3 i? One RCT found indeterminate, non-significant positive effects on CAOS and final exam scores (effect sizes 0.08 and 0.11). → bowen-ach

Evidence¶

lovett-caos¶

What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2020, January). Supporting Postsecondary Success intervention report: Open Learning Initiative (OLI). https://ies.ed.gov/ncee/wwc/InterventionReport/698

q3 · i2 · causal · r2

WWC calculations from Lovett, Meyer, & Thille (2008), an RCT at Carnegie Mellon University with 61 students in an introductory statistics course. Intervention mean "68.00 (18.90)" versus comparison 53.00 (30.00), effect size 0.55, improvement index +21, p = .044, statistically significant.

"Comprehensive Assessment of Outcomes in Statistics (CAOS) (Lovett, Meyer, & Thille, 2008)b Postsecondary students 61 students 68.00 (18.90) 53.00 (30.00) 15.00 0.55 +21 .044"

schunn-final¶

What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2020, January). Supporting Postsecondary Success intervention report: Open Learning Initiative (OLI). https://ies.ed.gov/ncee/wwc/InterventionReport/698

q2 · i? · causal · r2

WWC calculations from Schunn & Patchan (2009), a quasi-experimental study of a symbolic logic course at Carnegie Mellon with baseline equivalence established. Final exam means "62.35 (22.71)" versus 64.13 (25.68), effect size -0.07, improvement index -3, p = .547, not significant.

"Final exam score (Schunn & Patchan, 2009)c Postsecondary students 276 students 62.35 (22.71) 64.13 (25.68) -1.78 -0.07 -3 .547"

bowen-ach¶

What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2020, January). Supporting Postsecondary Success intervention report: Open Learning Initiative (OLI). https://ies.ed.gov/ncee/wwc/InterventionReport/698

q3 · i? · causal · r3

WWC calculations from the Bowen et al. (2014) RCT: final exam effect size 0.11, improvement index +4, p = .248; the CAOS outcome in the same study showed effect size 0.08, +3, p = .420. Both characterized as indeterminate.

"Final exam score (Bowen et al., 2014)a Postsecondary students 431 students 0.57 (0.13) 0.55 (0.22) 0.02 0.11 +4 .248"

Discussion¶

Related Claims¶

• OLI statistics students completed the semester's material in approximately half the regularly scheduled time — related
• OLI evidence base covers community college and postsecondary students across diverse racial and ethnic groups — related
• OLI has potentially positive effects on credit accumulation and persistence among postsecondary students — related
• Suh et al. (2019) found a positive but non-significant effect on course passing rate (effect size 0.80) and a negative non-significant effect on final exam score — related