A deployed reflection agent followed its prompt's countable rules but broke its dispositional rules about manner¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In a live deployment, the agent met checkable rules (72% of Study 2 replies within the two-sentence cap; scripted endings in 98% of sessions) while violating every dispositional rule, such as validating in 51.7% of turns despite an instruction not to be overly agreeable. → fidelity-audit

Evidence¶

fidelity-audit¶

Subigya K. Nepal, Serena Soh, Noah Vinoya, SoHyun Park, Mahnaz Roshanaei, and Gabriella Harari. 2027. Faithful Where It Can Be Checked: Auditing a Reflection Agent Against Its System Prompt in a Randomized Trial. In Proceedings of CHI Conference on Human Factors in Computing Systems (CHI '27). ACM, New York, NY, USA. https://arxiv.org/abs/2609.19635 doi:10.48550/arxiv.2609.19635

q2 · i? · associational · r2

Transcript audit of the deployed GPT-4o agent in the trial's Study 2, comparing behavior rates against the prompt's own rules. The agent met the countable rules — "72% of Study 2 replies fit the two-sentence cap" — while the same section reports validation in 51.7% of turns against a don't-overdo-praise instruction.

"The agent largely followed the countable rules: 72% of Study 2 replies fit the two-sentence cap, scripted endings appeared in 98% of sessions, and the pause and crisis protocols fired with their scripted wording"

Discussion¶

Related Claims¶

• Five general instruction-following evaluators could not reach deployable activation rates (best BMK 2.23%–20.72%) because fixed rubrics cannot enumerate long per-conversation instruction sets — related
• Monolithic single-prompt LLM architectures hinder responsible educational deployment by collapsing distinct pedagogical functions — related
• An explicit prompt instruction to challenge gently changed nothing: the agent challenged in about 1% of turns under both prompts — related
• The agent's praise was indiscriminate affirmation: validation followed about half of participant turns regardless of what the participant said — related
• State-of-the-art conversational agents (Blender and GPT-3) replied adequately to students but were well behind human performance in helping the student (reported from a dialogue study) — related
• Contrary to expectations, agent-led reflection sessions typically ended with participants sounding more certain of their plans than when they started — related