Distinguish learner research from user research and choose success metrics tied to educational outcomes rather than engagement¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The report's EnGen case study recommends that developers not conflate metrics meaningful in consumer tech with evidence of learning. As Dr. Brown puts it, "user research is conflated with learner research, and we shouldn't do that": clicks or time on page rarely paint the full picture in an edtech context, and teams should consult research on how to assess specific educational outcomes.

Design Implications¶

Context¶

Requirements¶

• Breaking the product idea into core components and reviewing literature on each
• Consulting research to identify assessments of the specific educational outcome targeted

Constraints¶

• The report acknowledges the best-fitting research may not exist; teams should keep improving research design as the product matures

Target Learners¶

• adult English language learners and other learners of edtech products

Target Learning Goals¶

• measuring genuine educational outcomes such as language acquisition rather than engagement proxies

Related Strategies¶

• Plan and design backwards from a research-based learning goal to features, metrics, and measurement tools
• Build formal feedback mechanisms and an evaluation plan into ed-tech pilots

Examples¶

-

Key Sources¶

• Van Nostrand, P., Noakes, S., Shah, Z., & Luke Luna C. (2022, February). An overlooked indicator of edtech quality: The use of learning sciences research. Digital Promise. https://doi.org/10.51388/20.500.12265/150