Immediate success rates are higher after injected bugs than after natural bugs¶
Claim · All claims
Evidence · 1 study · associationalr2·q3
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Immediate success for any action is 89.92% for i:first vs 51.16% for n:first, with the same contrast holding within prompt and edit actions. → rq1-success
Evidence¶
rq1-success¶
Victor-Alexandru Pădurean, Kaitlin Riegel, Alkis Gotovos, Jyotika Mahapatra, Ahana Ghosh, Paul Denny, Juho Leinonen, James Prather, and Adish Singla. 2026. When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code. Proceedings of the ACM Conference on International Computing Education Research Vol. 1 (ICER 2026 Vol. 1). https://doi.org/10.1145/3765964.3811667
q3 · i? · associational · r2
Log analysis of student sessions in the Prompt Programming platform. Immediate success is defined counterfactually for prompt actions using the injection audit, and by reaching a passing result through edit-and-run attempts for edit actions. Injected bugs show "success for any action is" substantially higher.
"In the 'first turn' summaries, success for any action is89 .92%fori:firstvs.51 .16%forn:first; the same contrast holds within prompt actions (87.74%vs.43 .30%) and edit actions (92.22%vs.63 .33%)."
Discussion¶
Related Claims¶
- Students make smaller code changes and fewer edit-and-run attempts after injected bugs than after natural bugs — related
- Injected bugs prompt more direct code edits while natural bugs prompt more reprompting in prompt-based programming tasks — related
- Prompting strategies differ by bug source: natural bugs elicit specification-oriented strategies while injected bugs elicit guidance-oriented strategies — related
- Initial prompting method did not explain immediate success on the third problem (P3) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
higher-immediate-success-after-injected-bugs |
title |
Immediate success rates are higher after injected bugs than after natural bugs |
description |
Immediate success rates are higher after injected bugs than after natural bugs |
status |
review |
evidence_strength |
moderate |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — rq1-success |
Frontmatter is the page's machine-readable record — see the Schema & Guide.