Solving a creative problem-solving task with ChatGPT is perceived as easier and requires less mental effort, but is not more interesting¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r3 · q3 · i2 medium
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 Task resolution with ChatGPT was perceived as moderately easier, F(1,142) = 12.16, η2p = .08 (d = 0.56 in Table 2). → Urban 2024 q3 i2 Task resolution with ChatGPT required moderately less mental effort, F(1,142) = 12.07, p < .001, η2p = .08 (d = 0.58 in Table 2). → Urban 2024 q3 i? Task interest did not differ between conditions, F(1,142) = 0.27, p = .602, η2p = .00 (d = 0.08). → Urban 2024 (2)

Evidence¶

Urban 2024¶

Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT Improves Creative Problem-Solving Performance in University Students: An Experimental Study. Computers & Education. https://doi.org/10.1016/j.compedu.2024.105031

q3 · i2 · causal · r3

ANCOVA on post-task perceived difficulty (100-point scale) in the randomized two-condition experiment. Table 2 reports difficulty d = 0.56 and effort d = 0.58 with experimental means of 37.08 and 40.08 versus control 50.22 and 54.07.

"task resolution with ChatGPT was perceived as moderately easier than task resolution without it, F(1,142) = 12.16, p = <.001, η2p = .08."

Urban 2024 (2)¶

Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT Improves Creative Problem-Solving Performance in University Students: An Experimental Study. Computers & Education. https://doi.org/10.1016/j.compedu.2024.105031

q3 · i? · causal · r3

ANCOVA on post-task task interest (100-point scale) in the same experiment; the article calls this a rather surprising finding given prior chatbot studies reporting higher interest. No equivalence test was reported.

"As illustrated in Figure 3, there were no significant differences between the control and experimental groups in task interest, F(1,142) = 0.27, p = .602, η2p = .00."

Discussion¶

Related Claims¶

• ChatGPT use boosts university students' self-efficacy for task resolution, but self-efficacy barely tracks actual performance — related
• ChatGPT use improves the quality, elaboration, and originality of university students' solutions to a complex ill-defined problem-solving task — related
• Self-reported mental effort increased less across tasks in the ChatGPT condition while perceived difficulty rose equally in both groups — related
• Effects of GenAI availability on self-reported understanding are insignificant; effects on interest are significant only under the transient-COVID assumption — related
• Prior knowledge and intervention condition predict repeated cognitive load and interest measures: prior knowledge lowers effort and difficulty and raises interest and understanding, and prompting groups perceive tasks as more difficult — related
• In Study 1, students rated the revised intervention as more interesting and more informative than the original intervention — related
• ChatGPT use does not change absolute self-evaluation accuracy, but self-evaluation judgments fail to correlate with performance under ChatGPT assistance — related