Dynamically generated natural-language feedback was not demonstrated to be superior to a non-NLP feedback condition that told students the answer¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? The article reports being unable to demonstrate that dynamically generated (ELICIT) feedback produced better outcomes than a TELL condition that immediately gave the correct answer without analysis. → Dzikovska 2014

Evidence¶

Dzikovska 2014¶

Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., & Campbell, G. (2014). BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics. International Journal of Artificial Intelligence in Education, 24, 284–332. https://doi.org/10.1007/s40593-014-0017-9

q3 · i? · causal · r2

System evaluation in which participants were randomly assigned to the ELICIT version (adaptive natural language feedback) or the TELL version (neutral acknowledgment plus correct answer); the only difference was feedback type. The article reports it was "unable to d emonstrate" the dynamic feedback's superiority. No effect size for this contrast is printed in the available text.

"However, we are unable to d emonstrate that dynamically generated feedback is superior to a non-NLP feedback condition."

Discussion¶

Related Claims¶

• Human-tutoring transcript analysis informed BEETLE II tactics, including defaulting to the full answer after three unsuccessful elicitation attempts — related
• Human tutoring has a medium effect over no tutoring, and in one small study students learned as well when tutors only prompted them as when tutors also explained and gave feedback — related
• Systematic prompting of an LLM can generate cognitively activating formative feedback aligned with research-based scaffold types (demonstrated, not yet empirically validated) — related
• The BEETLE II curriculum produces large pre- to post-test learning gains relative to a no-training control group (effect size 1.72) — related