In a four-wave MOOC intervention study, statistically significant first-wave findings failed to replicate in subsequent preregistered waves¶

Claim · All claims
Evidence · 1 study (2 entries) · theoretical r3 · q3
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? In a large-scale online learning experiment, significant effects found after revising exclusion criteria post hoc did not replicate when the study was re-run in a new wave under a new preregistration. → Preregistration and Registered Reports 2021 q3 i? A first-wave finding that the value relevance affirmation benefited racial minorities in the United States (N=96,548) was never confirmed in three subsequent waves (N of 102,969, 41,933, and 27,719). → Preregistration and Registered Reports 2021 (2)

Evidence¶

Preregistration and Registered Reports 2021¶

Preregistration and Registered Reports. (2021). https://osf.io/ya594

q3 · i? · theoretical · r3

Case study of a large randomized experiment in over 250 MOOCs from Harvard, MIT, and Stanford. After post hoc exclusion-criteria changes produced significant results, the team wrote a new preregistration and re-ran the study; the new wave "led to null results".

"Once again, our new predictions led to null results. The significant effects that we found with our modified exclusion criteria did not replicate in the new wave of the study."

Preregistration and Registered Reports 2021 (2)¶

Preregistration and Registered Reports. (2021). https://osf.io/ya594

q3 · i? · theoretical · r3

Within the same four-wave experiment, an exploratory subgroup finding in wave one (N=96,548) could not be predicted or replicated in waves of N=102,969, 41,933, and 27,719. The article reports this as an illustration of the garden of forked paths.

"In the first wave (N=96,548), for example, we found evidence that our value relevance affirmation benefited racial minorities in the United States, but we never confirmed this finding in the subsequent three waves (with N of 102,969, 41,933, and 27,719 respectively"

Discussion¶

Related Claims¶

• A brief online empathic-discipline exercise for teachers halved suspensions in one randomized field experiment, while its preregistered replication found a small, marginal overall effect but narrowed group disparities — related
• A brief values-affirmation writing exercise raised African American seventh-graders' grades in two small field experiments, but a large same-district replication found no effect — related
• MOOC video engagement declines with video length, with median engagement under half the duration for videos longer than nine minutes — related