Indirect prompt injections combining instruction manipulation, role-playing, and obfuscation altered AI-assigned grades with attack success rates of 94–100% while remaining undetected¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? In a red-team adversarial evaluation of Microsoft Copilot grading a synthetic failing essay, a prompt injection hidden behind an image layer succeeded in 17 of 18 iterations, and a revised end-of-document injection succeeded in 9 of 9 iterations, changing the grade without detection. → Niklas Humble 2026

Evidence¶

Niklas Humble 2026¶

Niklas Humble. (2026). Ethical Implications of Prompt Injection in AI-mediated Grading: An Adversarial Red-Team Evaluation. Journal of Academic Ethics. https://doi.org/10.1007/s10805-026-09758-9

q2 · i? · design · r2

Adversarial security evaluation (attempt 5): the attempt-3 injection paragraph re-used behind an image layer succeeded in 17 of 18 iterations, "with 17 successes and 1 failure", raising the grade to pass or pass with distinction without Copilot warning the user.

"In total, 18 iterations were conducted with the prompt injection hidden behind an image layer, with 17 successes and 1 failure"

Discussion¶

Related Claims¶

• Basic prompt injection attempts failed or were detected: instruction manipulation alone failed in all 9 iterations, role-playing without concealment was detected in output, and metadata-hidden injections failed in all 9 iterations — related
• Copilot's detection of one advanced injection did not persist: after reporting an embedded instruction once, the same file's injection succeeded in six subsequent iterations without warning — a narrower finding that bears on this claim
• Injected provider failures produced recorded unavailability and zero fabricated hints, while valid and reentrant conditions stored 30 synthetic hints each — related
• A revised injection combining instruction manipulation and role-playing with an explicit do-not-mention instruction succeeded in all 9 iterations across docx, pdf, and htm formats — possibly the same claim (merge candidate)
• Asking the AI tool directly about hidden instructions exposed the prompt injection in all 6 security-check iterations, but the authors caution this is not a robust safeguard — related
• Prompt injection attacks on agents (OpenClaw walletdrain, EchoLeak) bypass the user entirely, and no public AI literacy curriculum addresses this vector — related
• Deterrence layers failed because seeds changed parameters rather than task structure, scaffolding exposed solution steps, and hidden grading verified output consistency rather than authorship or understanding — related
• Learners rarely detect manipulation of offloaded information and may replace internal memory with altered external content — related