Copilot's detection of one advanced injection did not persist: after reporting an embedded instruction once, the same file's injection succeeded in six subsequent iterations without warning¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? After Copilot once detected and reported an embedded instruction in a pdf, six further iterations with the same file succeeded in changing the grade to pass with distinction without any reference to the hidden injection, which the authors argue could create a false sense of security. → Niklas Humble 2026

Evidence¶

Niklas Humble 2026¶

Niklas Humble. (2026). Ethical Implications of Prompt Injection in AI-mediated Grading: An Adversarial Red-Team Evaluation. Journal of Academic Ethics. https://doi.org/10.1007/s10805-026-09758-9

q2 · i? · causal · r1

Adversarial evaluation attempt 5, iterations 13–18: following the single detection event, "the same prompt injection was iterated additionally 6 times with the same file, with all successfully changing the grade" undetected. The authors warn the detection message could lull users into false security.

"However, after this statement the same prompt injection was iterated additionally 6 times with the same file, with all successfully changing the grade to pass with distinction (VG) without Copilot referencing the hidden prompt injection or warning the user"

Discussion¶

Related Claims¶

• Basic prompt injection attempts failed or were detected: instruction manipulation alone failed in all 9 iterations, role-playing without concealment was detected in output, and metadata-hidden injections failed in all 9 iterations — related
• Asking the AI tool directly about hidden instructions exposed the prompt injection in all 6 security-check iterations, but the authors caution this is not a robust safeguard — related
• A revised injection combining instruction manipulation and role-playing with an explicit do-not-mention instruction succeeded in all 9 iterations across docx, pdf, and htm formats — related
• Indirect prompt injections combining instruction manipulation, role-playing, and obfuscation altered AI-assigned grades with attack success rates of 94–100% while remaining undetected — a broader claim this one bears on
• Learners rarely detect manipulation of offloaded information and may replace internal memory with altered external content — related