For the Giving Effective Praise lesson, GPT-4-rephrased responses were rated significantly more accurate than human-expert rephrasings (p < 0.001), with no significant difference in responsiveness¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Accuracy scores of GPT-4 rephrased responses (zero-shot and few-shot) were significantly higher than human rephrased responses for Giving Effective Praise (p < 0.001), with median accuracy 1 versus -1 for humans. → Jionghao Lin 2024 q2 i? No significant difference was observed in responsiveness between GPT-4 and human rephrased responses for Giving Effective Praise. → Jionghao Lin 2024 (2)

Evidence¶

Jionghao Lin 2024¶

Jionghao Lin, Zifei Han, Danielle R. Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, Kenneth R. Koedinger. (2024). How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses. arXiv preprint. https://arxiv.org/abs/2405.00970

q2 · i? · design · r2

Mann-Whitney U tests on expert Likert ratings of rephrased responses for the Giving Effective Praise lesson; GPT-4 median accuracy was 1 versus human median -1 (Table 8 mean accuracy 0.92 few-shot vs -0.36 human).

"the accuracy scores of the rephrased responses generated using both GPT models (zero-shot and few-shot) were significantly higher than the responses rephrased by the humans ( p < 0.001)"

Jionghao Lin 2024 (2)¶

Jionghao Lin, Zifei Han, Danielle R. Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, Kenneth R. Koedinger. (2024). How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses. arXiv preprint. https://arxiv.org/abs/2405.00970

q2 · i? · design · r2

Responsiveness ratings for the same lesson showed no significant GPT-4 versus human difference; human rephrasings had higher variance (Table 8 mean responsiveness 0.44 for both few-shot and human).

"we did not observe a significant difference between the responsiveness score of the GPT-4 rephrased responses and human rephrased responses; however, the human rephrased responses had a higher variance"

Discussion¶

Related Claims¶

• For Reacting to Errors, GPT-4 rephrasings were rated more accurate than human rephrasings (few-shot p < 0.01, zero-shot p < 0.05) with no significant responsiveness difference — related
• For Determining What Students Know, no significant difference in accuracy or responsiveness was found between GPT-4 and human rephrasings — related