At publication, a frontier coding model could produce solutions to six of the ten research-shaped assignments but struggled with four, which the authors treat as non-durable design experience¶
Claim · All claims
Evidence · 1 study · designr2·q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q1 i? Codex 5.6-Sol could produce solutions to HW1, HW4, HW5, HW6, HW7, and HW8, but struggled with HW2, HW3, HW9, and HW10; the authors argue educational value must lie in required reasoning and checking rather than remaining unpromptable. → M. Mikhasenko 2026
Evidence¶
M. Mikhasenko 2026¶
M. Mikhasenko, M. Stahl, I. Segal, D. Parmar, A. Zimmer, and A. Kazatsky. (2026). AI in Particle Physics Education: Research Problems and Foundational Skills. https://arxiv.org/abs/2609.05719 doi:10.48550/arxiv.2609.05719
q1 · i? · design · r2
Instructor observation reported without a systematic benchmark: "Codex 5.6-Sol can produce solutions to HW1, HW4, HW5, HW6, HW7, and HW8" while struggling with HW2, HW3, HW9, and HW10. The article states no systematic benchmark of named models, prompts, or access tiers was conducted.
"At the time of publica- tion, Codex 5.6-Sol can produce solutions to HW1, HW4, HW5, HW6, HW7, and HW8. It struggles with HW2 (a complex multistep workflow), HW3 (a visual interface), HW9 (human input for visualization is needed), and HW10 (no multi-speaker podcast capability)."
Discussion¶
Related Claims¶
- Course design created frictions: undeclared programming prerequisites, copied AI-generated tutorial solutions, intimidating blackboard turns, and AI-guessing cycles for students lacking evaluation knowledge — related
- The workshop argues AI-resistant assignments are temporary at best, reframing design goals toward tasks whose difficulty survives AI completing part of the work — a broader claim this one bears on
- AI solutions can automate routine tasks but cannot yet handle the complex tasks of higher learning — related
- Three recurring AI failure modes arose in this project-based learning context: plausible-but-incorrect code, missing specialized knowledge, and limited long-term context — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
kt1-model-resistance-six-of-ten-assignments |
title |
At publication, a frontier coding model could produce solutions to six of the ten research-shaped assignments but struggled with four, which the authors treat as non-durable design experience |
description |
At publication, a frontier coding model could produce solutions to six of the ten research-shaped assignments but struggled with four, which the authors treat as non-durable design experience |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — m-mikhasenko-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.