Skip to content

Planned Comparable-Group Courseware Impact Study

Research Method · All research methods
Evidence · 2 claims (1 for, 1 mixed) · 1 study (1 design), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description

The article distills lessons from the NGCC evaluation about conducting impact studies: measuring impact requires an objective outcome measure, a comparison group, evidence of comparability, adequate numbers in both groups, and holding other influences constant. It warns that "Comparison groups and data sources need to be planned in advance if you want to get an apples-to-apples comparison of different versions of a course." It also cautions that busy technology developers, marketers, or faculty cannot be expected to design and execute rigorous impact studies without considerable help.

Accounts

  • Plan comparison groups and data sources in advance, because rigorous courseware impact studies require comparable treatment and comparison groups and cannot be left to busy developers or faculty alone.: The article distills lessons from the NGCC evaluation about conducting impact studies: measuring impact requires an objective outcome measure, a comparison group, evidence of comparability, adequate numbers in both groups, and holding other influences constant. It warns that "Comparison groups and data sources need to be planned in advance if you want to get an apples-to-apples comparison of different versions of a course." It also cautions that busy technology developers, marketers, or faculty cannot be expected to design and execute rigorous impact studies without considerable help. (Barbara Means et al. (2004))
  • Include comparison groups in edtech pilots whenever possible to generate interpretable efficacy results: This principle holds that pilot studies of edtech products should include a comparison group of students not using the tool so that outcome differences can be attributed to the tool. The article's two pilots illustrate the contrast: Cajon Valley's comparison-group design supported a significant finding, while Vista's single-group design left results uninterpretable. As the report states, districts' results "demonstrate the necessity to include comparison groups when possible in pilots in order to generate powerful results." (Luke et al. (2017))

Claims

-

Key Sources

  • Barbara Means and Julie Neisler. (2004). Continuous Improvement for Student Success. Digital Promise Global. https://www.newyorker.com/magazine/2004/12/06/the-bell-curve
  • Luke, Christina C.; Gibson, Sierra; Francisco, Aubrey T. (2017). Rapid Cycle Pilots: Using Edtech to Support English Learners. Digital Promise. https://www.digitalpromise.org. doi:10.51388/20.500.12265/26
Page metadata
Field Value
type research-method
id planned-comparable-group-courseware-impact-study
title Planned Comparable-Group Courseware Impact Study
description The article distills lessons from the NGCC evaluation about conducting impact studies: measuring impact requires an objective outcome measure, a comparison group, evidence of comparability, adequate numbers in both groups, and holding other influences constant.
status review
generated by process:settle-candidates on 2026-10-09
trust unverified — no one has confirmed this page's content

Frontmatter is the page's machine-readable record — see the Schema & Guide.