B2 problems required more effort than B1 problems, with lower success rates and greater variance in conversations, messages, and code executions¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Descriptive statistics indicate B2 problems required more effort than B1, with greater variance in conversations, messages, and code executions, and the article attributes this to B2 problems being harder given the lower success rate. → Victor-Alexandru Pădurean 2026

Evidence¶

Victor-Alexandru Pădurean 2026¶

Victor-Alexandru Pădurean, Kaitlin Riegel, Gweneth Barbre, Musa Blake, Paul Denny, Alkis Gotovos, Juho Leinonen, Stephen MacNeil, James Prather, and Adish Singla. (2026). Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks. Proceedings of the ACM Conference on International Computing Education Research Vol.1 (ICER 2026 Vol. 1). https://doi.org/10.1145/3765964.3811639

q2 · i? · associational · r1

Descriptive statistics (Table 1) for students completing B1 (N= 887) and B2 (N= 876), showing higher means and variance for messages and code executions in B2; Table 2 shows success rates falling to 0.44 on B2-3. No inferential tests were run because samples were not independent.

"Overall, the results align with B2 problems requiring more ef-fort, with greater variance in the variables (though, notably, there is a large variation in the number of messages sent for both batches)."

Discussion¶

Related Claims¶

• When prompts failed, students most commonly reported clarifying intent or specifying more detail, and reflecting on the problem depiction, rather than tracing code or examining test cases — related
• Strategy use showed no significant relationship with problems solved in B2, except writing code as prompt (r = .20), based on only twelve students — related
• Correlations between the number of prompt mistakes and number of messages sent were non-significant or weak, giving little evidence that mistake count influenced prompting effort — related