Self-administered Exam Playbook use predicts higher exam scores in real-world college classes (d = 0.18)¶

Claim · All claims
Evidence · 1 study (4 entries) · associational r2–r3 · q3 · i0 negligible
Currency · newest study 2022 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i0 Across 14 classes, students who used the Exam Playbook at least once scored on average 2.17 percentage points higher (Cohen's d = 0.18) than non-users. → Chen 2022 q3 i0 At the exam level within classes, users on a given exam scored 2.91 percentage points higher (d = 0.22) than non-users on that exam. → Chen 2022 (2) q3 i0 Controlling for college entrance exam scores, users scored 1.65 percentage points higher (d = 0.14) than non-users. → Chen 2022 (3) q3 i0 Excluding the Introductory Statistics course for which the intervention was designed, users still scored 1.60 percentage points higher (d = 0.13) than non-users. → Chen 2022 (4)

Evidence¶

Chen 2022¶

Chen, P., Teo, D. W. H., Foo, D. X. Y., Derry, H. A., Hayward, B. T., Schulz, K. W., Hayward, C., McKay, T. A., & Ong, D. C. (2022). Real-world effectiveness of a social-psychological intervention translated from controlled trials to classrooms. npj Science of Learning 7:20. https://doi.org/10.1038/s41539-022-00135-w

q3 · i0 · associational · r2

Random-effects mini meta-analysis across 14 classes (N = 12,065), treating each class as a separate experiment, found "2.17 ([95% CI: 1.13, 3.21], p < 0.001) percentage points higher", translating to Cohen's d = 0.18; the effect was positive in 13 of 14 classes.

"Our meta-analysis, summarized in Fig. 1, revealed that students who used the Exam Playbook in their class scored 2.17 ([95% CI: 1.13, 3.21], p < 0.001) percentage points higher than non-users, for their average exam score"

Chen 2022 (2)¶

Chen, P., Teo, D. W. H., Foo, D. X. Y., Derry, H. A., Hayward, B. T., Schulz, K. W., Hayward, C., McKay, T. A., & Ong, D. C. (2022). Real-world effectiveness of a social-psychological intervention translated from controlled trials to classrooms. npj Science of Learning 7:20. https://doi.org/10.1038/s41539-022-00135-w

q3 · i0 · associational · r2

Robustness check at the exam level across all 40 exams showed users "scored an average of 2.91" percentage points higher on that exam (d = 0.22, p < 0.001).

"A mixed-effects meta-analysis (with exam as a fixed effect within each class, and class as a random effect) across all 40 exams observed showed that students who used the Exam Playbook on a given exam scored an average of 2.91 ([1.81, 4.01], Cohen 's d = 0.22, p < 0.001) percentage points higher"

Chen 2022 (3)¶

Chen, P., Teo, D. W. H., Foo, D. X. Y., Derry, H. A., Hayward, B. T., Schulz, K. W., Hayward, C., McKay, T. A., & Ong, D. C. (2022). Real-world effectiveness of a social-psychological intervention translated from controlled trials to classrooms. npj Science of Learning 7:20. https://doi.org/10.1038/s41539-022-00135-w

q3 · i0 · associational · r3

Robustness check controlling college entrance exam scores as a covariate found users "scored an average of 1.65 ([0.55, 2.75], Cohen 's d = 0.14, p = .003) percentage points higher".

"Exam Playbook users scored an average of 1.65 ([0.55, 2.75], Cohen 's d = 0.14, p = .003) percentage points higher than non-users on their average exam score."

Chen 2022 (4)¶

Chen, P., Teo, D. W. H., Foo, D. X. Y., Derry, H. A., Hayward, B. T., Schulz, K. W., Hayward, C., McKay, T. A., & Ong, D. C. (2022). Real-world effectiveness of a social-psychological intervention translated from controlled trials to classrooms. npj Science of Learning 7:20. https://doi.org/10.1038/s41539-022-00135-w

q3 · i0 · associational · r2

Conservative generalizability test excluding Introductory Statistics (12 classes) found users scored "an average of 1.60 ([1.00, 2.19], d = 0.13, p < 0.001) percentage points higher"; controlling entrance scores reduced it to 1.07 (d = 0.09).

"students who used the Exam Playbook scored an average of 1.60 ([1.00, 2.19], d = 0.13, p < 0.001) percentage points higher than non-users. When controlling for college entrance exam scores, we observed a 1.07 percentage points difference ([0.29, 1.85], d = 0.09, p = 0.007)."

Discussion¶

Related Claims¶

• Within individuals, adopting the Exam Playbook tracks better subsequent exam performance and dropping it tracks worse — related
• Class climate—peer uptake and instructor credit incentives—relates to larger Exam Playbook effect sizes — related
• Exam Playbook benefits relate to dosage and to earlier use before exams — related
• Exam Playbook benefits were larger for female users and for first-generation students, reducing achievement gaps — related
• A strategic resource-use exam playbook intervention raised final course grades in introductory statistics across two randomized cohorts, with a dosage effect for completing it twice — related