User-Written AI Agent Rules Fail to Stop Overreach, Study Finds

A new study of 113 non-technical participants reveals a counterintuitive finding: letting users write their own permission policies for AI agents provides weaker protection against unwanted actions than either human-in-the-loop approval or automated model review. Participants who set rules in advance blocked 20 percentage points less overreach than those who approved each action individually — not because the rules were poorly designed, but because users overwhelmingly chose 'ask me' over 'never allow,' effectively routing most decisions back to runtime anyway. The research exposes a fundamental tension in agentic AI governance: the gap between what users say they want in the abstract and what they actually permit in the moment.

The Promise and Problem of User-Authored Permission Policies

As AI agents take on increasingly autonomous roles — managing email, files, payments, and personal data — the question of how non-technical users can meaningfully control agent behavior has become critical. One appealing solution is letting users write standing permission rules in plain language: define categories of consequences, then set blanket 'allow,' 'ask,' or 'never' rules for each. Researchers from [institution] tested this approach head-to-head against two alternatives, and the results should give AI product designers pause.

The study, Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?, enrolled 113 participants without professional software backgrounds across three conditions: per-action human-in-the-loop approval (HITL), automated model-based review (AUTO), and user-authored consequence policies (POLICY). All participants then supervised a simulated 18-action agent day that embedded 7 deliberate overreach actions — tasks the agent performed beyond what users had actually requested.

Key Findings

Policy rules blocked significantly less overreach. The POLICY condition blocked 20.1 percentage points fewer overreach actions than HITL (95% CI: −32.1 to −8.1) and 14.5 points fewer than AUTO (95% CI: −25.8 to −3.2).
Users almost always chose 'ask,' not 'never.' Of 140 rules set by POLICY participants, 114 (81%) were set to 'ask,' meaning users opted to preserve case-by-case judgment rather than commit to a blanket policy. Only a handful of actions were categorically blocked.
Most overreach happened with user approval. Of the 148 overreach actions executed in the POLICY condition, 133 occurred after explicit user approval at runtime. Only 15 ran automatically under 'allow' rules — meaning the policy itself wasn't the primary failure mode; human approval at the moment of action was.
Runtime burden reduction was minimal when setup time was included. POLICY did reduce runtime prompts from 18.0 to 10.9 on average, but when rule-setup time was factored in, total intervention time was not reliably lower.
POLICY users approved overreach at the highest rate. Across all 7 overreach action types, POLICY participants had the highest approval rate — suggesting that the act of setting rules may create a false sense of security that lowers vigilance at runtime.

The Preference-Commitment Gap

The study's most important contribution may be conceptual. The authors describe a preference-commitment gap: users express preferences about how they want an agent to behave, but those preferences don't translate into binding commitments that actually constrain behavior. Choosing 'ask me' for a rule category feels like responsible governance, but it simply defers the decision — and at the moment of decision, users tend to say yes.

This has direct implications for AI agent product design. Mechanisms that feel like control (rule editors, permission dashboards, category-based policies) may not deliver control if users reliably approve actions at runtime regardless of their stated preferences. The research suggests that the design of how users are prompted at runtime — not just whether they set rules in advance — is where protection is actually won or lost.

Implications for Practitioners Building Agentic Systems

Don't rely on user-authored rules as a primary safety layer. Standalone policy editors are insufficient for preventing overreach if users approve exceptions at runtime. Complement them with automated guardrails or conservative defaults.
Design runtime prompts carefully. Since most overreach in the POLICY condition was user-approved, the framing, context, and friction of runtime approval dialogs matters enormously. Consider making overreach-risk salient at the moment of approval.
Reconsider 'ask' as a default rule option. Giving users an easy 'ask me every time' option may undermine the purpose of a standing policy. Designs that nudge users toward genuine commitment (e.g., requiring explicit justification for 'ask' selections) could improve real-world protection.
Automated model review (AUTO) performed well. The AUTO condition — where a model reviewed each action without user involvement — outperformed user-authored policies on overreach blocking. Hybrid approaches that combine model-level screening with selective human escalation may offer a better balance than user rule-writing alone.
Measure actual protection, not perceived control. User satisfaction with a permission system is not a proxy for safety. Evaluation frameworks for agentic AI should include behavioral overreach metrics, not just usability scores.

Bottom Line

This study offers a sobering empirical check on an intuitively appealing design pattern. Giving users a policy editor feels like empowerment, but when users fill it with 'ask me' rules and then say yes at runtime, the policy provides little more than the illusion of control. For practitioners building agentic systems where real money, data, or actions are at stake, these findings argue for treating user-authored policies as one layer in a defense-in-depth strategy — not as a sufficient control mechanism on their own.