ChatGPT use does not change absolute self-evaluation accuracy, but self-evaluation judgments fail to correlate with performance under ChatGPT assistance¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r3 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? No between-group difference in absolute self-evaluation accuracy for quality, F(1,142) = 0.16, or originality, F(1,142) = 1.08. → Urban 2024 q3 i? With ChatGPT, self-evaluation judgments showed no correlation with solution quality (r = .02) or originality (r = -.03), suggesting participants could not calibrate their judgments. → Urban 2024 (2)

Evidence¶

Urban 2024¶

Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT Improves Creative Problem-Solving Performance in University Students: An Experimental Study. Computers & Education. https://doi.org/10.1016/j.compedu.2024.105031

q3 · i? · causal · r3

ANCOVA on bias indices (self-evaluation minus transformed performance) in the randomized experiment (N = 145). The article reports no statistically significant group difference; equivalence was not tested.

"there were no differences between the control and experimental group in self-evaluation accuracy, nor in self-assessment of quality, F(1,142) = 0.16, p = .687, η2p = .00, or originality, F(1,142) = 1.08, p = .301, η2p = .01."

Urban 2024 (2)¶

Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT Improves Creative Problem-Solving Performance in University Students: An Experimental Study. Computers & Education. https://doi.org/10.1016/j.compedu.2024.105031

q3 · i? · causal · r3

Exploratory examination of the correlation matrix in Table 3 from the same experiment. The article cautions that absence of evidence is not evidence of absence and that aggregated data may hide different impacts on different participants.

"With the use of ChatGPT, there is no correlation between self-evaluation judgments and quality ( r = .02) or originality ( r = -.03) of solutions on PIT, suggesting that participants at the group level were unable to calibrate their judgments according to their performance."

Discussion¶

Related Claims¶

• ChatGPT use boosts university students' self-efficacy for task resolution, but self-efficacy barely tracks actual performance — related
• ChatGPT use improves the quality, elaboration, and originality of university students' solutions to a complex ill-defined problem-solving task — related
• Self Assessment Accuracy Is Low Without Training — related
• Free ChatGPT use boosts originality on the assisted creative task but the advantage disappears when assistance is removed — related
• Providing a rubric during self-assessment improves the absolute accuracy of high school students' self-assessments of written scientific abstracts — reports the opposite
• Learning-process and psychological constructs correlate only weakly with aggregate final artistic performance — related
• Solving a creative problem-solving task with ChatGPT is perceived as easier and requires less mental effort, but is not more interesting — related
• Expert example standards improve university students' judgment accuracy (bias, absolute accuracy, and relative accuracy) when evaluating self-generated examples — related