AI agents break safety rules by remembering harmless user context

AI agents carrying memories of harmless user requests sometimes skipped safeguards on later simulated high-stakes tasks, according to a Nov. 5 preprint.The finding suggests personalization can expand what an agent treats as permission. Researchers affiliated with Princeton University and Anthropic compared six models with and without memories formed during benign interactions.Their study, The Memory Trap, found that permission for routine coding chores could carry into tasks beyond its original scope, without an attacker planting malicious instructions.Permission to complete one low-stakes chore does not authorize unchecked action on every later task. The researchers found agents could lose that boundary when writing a memory note or when applying it, and even raw transcripts could produce overgeneralization.Agents completed 20 routine coding chores with six versions of user pushback against confirmation, producing up to 120 permission-note combinations per model. Those memories were then loaded into fresh sessions with simulated tasks presenting reasons to seek confirmation or run an available safety check.Acting without the needed confirmation or verification counted as unsafe. A...