Perceived usefulness of ChatGPT, an invalid cue, is associated with overestimation of performance¶

Claim · All claims
Evidence · 1 study · causal r3 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Perceived usefulness of ChatGPT was moderately associated with higher bias index (overestimation), median r = .24, while not correlating with actual performance, median r = -.01. → Urban 2024 q2 i? Perceived task difficulty was negatively associated with self-evaluation accuracy: participants who found the task easier overestimated more. → Urban 2024

Evidence¶

Urban 2024¶

Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT Improves Creative Problem-Solving Performance in University Students: An Experimental Study. Computers & Education. https://doi.org/10.1016/j.compedu.2024.105031

q2 · i? · causal · r3

Exploratory correlational analysis within the experimental group (N = 77) reported in the Results. The article concludes the more useful participants found ChatGPT, the more they overestimated their answers' quality and originality, and labels these results exploratory.

"the perceived usefulness of ChatGPT is moderately associated with higher bias index (i.e. overestimation) of performance (median r = .24). This may prove crucial because perceived usefulness of ChatGPT does not correlate with actual problem-solving performance (median r = -.01)."

Discussion¶

Related Claims¶

• ChatGPT use boosts university students' self-efficacy for task resolution, but self-efficacy barely tracks actual performance — related
• Teachers' self-reported AI literacy correlates only weakly with their objectively measured performance (r = 0.07–0.24) — a broader claim this one bears on
• Students' self-reported perception of peer assessment's benefit is virtually uncorrelated with their actual measured benefit — a broader claim this one bears on
• Students' self-reported AI literacy was moderately negatively correlated with objectively measured AI competency, suggesting potential overestimation of AI abilities — related