For Reacting to Errors, GPT-4 rephrasings were rated more accurate than human rephrasings (few-shot p < 0.01, zero-shot p < 0.05) with no significant responsiveness difference¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For Reacting to Errors, few-shot rephrased responses were significantly more accurate than human rephrasings (p < 0.01), and even zero-shot rephrasings were more accurate than human alterations (p < 0.05). → Jionghao Lin 2024 q2 i? No significant difference in responsiveness between GPT-revised and human-revised responses for Reacting to Errors. → Jionghao Lin 2024 (2)

Evidence¶

Jionghao Lin 2024¶

Jionghao Lin, Zifei Han, Danielle R. Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, Kenneth R. Koedinger. (2024). How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses. arXiv preprint. https://arxiv.org/abs/2405.00970

q2 · i? · design · r2

Mann-Whitney U tests on expert ratings for the Reacting to Errors lesson; GPT-4 median accuracy was 1 versus human median 0 (Table 9 mean accuracy 0.86 few-shot vs 0.07 human).

"the accuracy of responses rephrased using the few-shot approach was significantly higher than those rephrased by humans (p< 0.01). Even the zero-shot rephrased responses were more accurate than human alterations (p< 0.05)"

Jionghao Lin 2024 (2)¶

Jionghao Lin, Zifei Han, Danielle R. Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, Kenneth R. Koedinger. (2024). How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses. arXiv preprint. https://arxiv.org/abs/2405.00970

q2 · i? · design · r2

Responsiveness ratings for Reacting to Errors clustered between 0 and 1 for both GPT and human revisions; average word count per response remained consistent between GPT and human revisions.

"most of the scores from the GPT-revised and human-revised responses were clustered between 0 and 1, with no significant difference in responsiveness between them"

Discussion¶

Related Claims¶

• Few-shot learning with increasing shots generally improves GPT-4 classification performance, with the 5-shot variant strongest for Reacting to Errors (F1 0.867, AUC 0.866, TNR 0.83) — related
• For the Giving Effective Praise lesson, GPT-4-rephrased responses were rated significantly more accurate than human-expert rephrasings (p < 0.001), with no significant difference in responsiveness — related
• For Determining What Students Know, no significant difference in accuracy or responsiveness was found between GPT-4 and human rephrasings — related