Learners receiving ChatGPT-generated hints showed statistically significant learning gains compared to a no-hint control in an online mathematics tutoring experiment¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2024 · checked against its sources 2026-10-17 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q3 i? The ChatGPT hint condition produced statistically significantly greater learning gains than the no-hint control (Dunn's test p = 0.011), with a 17% gain versus the control's 1.85%. → Pardos ZA 2024

Evidence¶

Pardos ZA 2024¶

Pardos ZA, Bhandari S (2024) ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLoS ONE 19(5): e0304013. https://doi.org/10.1371/journal.pone.0304013

q3 · i? · causal · r2

Randomized online experiment (N = 274 Mechanical Turk participants) with a 3x4 design in the OATutor system; post-hoc Dunn's test on ranked learning gains. The article reports "statistically significantly greater learning gains (p = 0.011)" for ChatGPT hints over control; no observed effect size is printed for this comparison (the Cohen's d of 0.5 in the article is the effect size assumed for its power analyses).

"From this test, we found that compared to the 1.85% gain of the control condition (i.e., “no hints”), learners in the ChatGPT condition exhibited statistically significantly greater learning gains (p = 0.011)."

Studies bearing on this claim¶

Do hints given on demand at the step where the learner is stuck improve learning over working without them? This table gathers the studies behind this claim and the 10 narrower claims that bear on it: 7 distinct studies (3 causal, 2 design, 1 review, 1 associational), q2–q3, 0 of 7 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Learning gains from ChatGPT-generated hints were not statistically significantly different from human tutor-authored hints, though ChatGPT gains were numerically higher in all subjects — related
• Both hint conditions took more time-on-task than control, but ChatGPT and human tutor conditions did not differ in session time — related
• Learning gains showed significant main effects of hint condition and subject, with no significant condition-by-subject interaction — a broader claim this one bears on
• Students who teach a ChatGPT teachable agent in natural language achieve greater knowledge gains than students who learn the same material from online videos — related
• AI multimodal feedback produces post-test learning gains equivalent to educator feedback after controlling for pre-test performance — related
• Students receiving instructor feedback showed significantly greater improvements in lab assignment scores than students receiving AI-generated feedback, after controlling for initial knowledge — related
• AI-generated homework hints were positively associated with exam performance only when exam questions closely matched the homework content — related
• Feedback source (human tutor vs GPT-4) had no significant effect on ENL students' posttest writing scores — related
• AIfred and ChatGPT produce comparable math scores while assistance is available — related
• K-12 learners preferred incremental hints and multiple attempts over immediately revealed solutions — related
• Generative AI-generated hints can be as effective as human-generated hints for mathematics performance, but ensuring accuracy requires validation procedures that discard a substantial share of generated hints — a broader claim this one bears on