A regression discontinuity design on Review Exercise data supports a causal link between leveling up and higher assessment accuracy, at least for the first level-up¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because 2 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q2 i? Learners who studied a Review Exercise source lesson twice (Level 0 and Level 1) were more likely to answer correctly than learners who had studied the source lesson once but the preceding lesson twice. → Lucy Portnoff 2021

Evidence¶

Lucy Portnoff 2021¶

Lucy Portnoff, Erin Gustafson, Klinton Bicknell and Joseph Rollinson. (2021). Methods for Language Learning Assessment at Scale: Duolingo Case Study. Proceedings of The 14th International Conference on Educational Data Mining (EDM21). https://educationaldatamining.org/edm2021/

q2 · i? · causal · r2

Logistic regression with a regression discontinuity design on approximately two months of Review Exercise data from all Duolingo courses compared learners at the same-vs-lower level cut-off for the source lesson, controlling for self-selection. The article reports the Level 1:Same Level coefficient was significantly higher; no effect size is printed.

"We can see that Level 1:Same Level is significantly higher than Level 0:Lower Level. This effect indicates that learners who have studied a Review Exercise source lesson twice (at Level 0 and Level 1) are more likely to provide a correct response"

Discussion¶

Related Claims¶

• Completing more skill levels is associated with higher Checkpoint Quiz post-test accuracy, with positive coefficients for Levels 1, 2, and 4 — related
• Leveling up lessons preceding the source lesson improves Review Exercise accuracy, indicating transfer of learning benefits across lessons within a skill — related
• Elaborative interrogation produced lower posttest accuracy than reading the lesson twice in a written lesson on posterior probability — related
• Diagrams improved posttest accuracy only in the read-twice condition, not in the questioning conditions — related
• Robust RDD methods protect against model misspecification but require much larger sample sizes to detect small effects — related
• A single practice test produced a larger effect (g = 0.70) than two or more practice tests (g = 0.51) — related