Checkpoint Quiz: a unit-terminal achievement test with embedded pre-test/post-test design¶

Element · All elements
Evidence · 1 claim (1 for) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

The Checkpoint Quiz is a custom-built assessment learners complete when finishing a unit at a Checkpoint. It is an "achievement test that measures the extent to which our learners have achieved the objectives for each unit of a course", with items independent from course exercises so learners cannot study the test items. Each randomly generated quiz has 15 items: seven pre-test items for the upcoming unit, seven post-test items for the completed unit (the same items seen as pre-test), and one self-directed writing item. This pre-test/post-test design establishes a baseline so gain in accuracy can be assessed. Items cover vocabulary, grammar, listening, reading, and free-form writing, with curated tags for grammatical concepts and communicative components.

Design Implications¶

Context¶

Requirements¶

• Items must be independent from course exercise items to preserve test validity
• Items designed by curriculum experts with quality analyses by assessment scientists

Constraints¶

• Data collection is slow, occurring only at Checkpoints
• Available only for a subset of Duolingo courses

Target Learners¶

• Duolingo language learners on mobile apps and website

Target Learning Goals¶

• Unit-level achievement of vocabulary, grammar, listening comprehension, reading comprehension, and writing objectives

Claims¶

• Completing more skill levels is associated with higher Checkpoint Quiz post-test accuracy, with positive coefficients for Levels 1, 2, and 4 [+W]

Related Elements¶

• Review Exercises: injected review items sampled from earlier skills for scalable, fine-grained assessment

Examples¶

-

Key Sources¶

• Lucy Portnoff, Erin Gustafson, Klinton Bicknell and Joseph Rollinson. (2021). Methods for Language Learning Assessment at Scale: Duolingo Case Study. Proceedings of The 14th International Conference on Educational Data Mining (EDM21). https://educationaldatamining.org/edm2021/