For Determining What Students Know, no significant difference in accuracy or responsiveness was found between GPT-4 and human rephrasings¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Evaluation of rephrased responses for Determining What Students Know revealed no significant difference in accuracy and responsiveness across GPT-4 zero-shot, few-shot, and human approaches. → Jionghao Lin 2024

Evidence¶

Jionghao Lin 2024¶

Jionghao Lin, Zifei Han, Danielle R. Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, Kenneth R. Koedinger. (2024). How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses. arXiv preprint. https://arxiv.org/abs/2405.00970

q2 · i? · design · r2

Mann-Whitney U tests on expert ratings for the Determining What Students Know lesson (Table 10 mean accuracy 0.70 few-shot vs 0.85 human); equivalence was not established, only no detected difference. Few-shot responsiveness was marginally higher than human (p = 0.08).

"our evaluation of the rephrased responses from the lesson Determining What Students Know, as illustrated in Fig. 5 and Table 10, revealed no significant difference in the dimensions of accuracy and responsiveness across the three approaches"

Discussion¶

Related Claims¶

• For the Giving Effective Praise lesson, GPT-4-rephrased responses were rated significantly more accurate than human-expert rephrasings (p < 0.001), with no significant difference in responsiveness — related
• Classification was weakest for the Determining What Students Know lesson, where zero-shot learning scored F1 0.66 and AUC 0.668 — related
• For Reacting to Errors, GPT-4 rephrasings were rated more accurate than human rephrasings (few-shot p < 0.01, zero-shot p < 0.05) with no significant responsiveness difference — related