Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

Teach AI agents when to stop

Smart AI agents need boundaries that tell them when to act, when to escalate, and when to stop.

[Photo: Getty Images]

Profile photo of Neel Somani

Founder · Eclipse Labs

Imagine an AI agent inside your company’s tools, clicking through an interface, moving files, approving a request, or preparing to send money. Now imagine the AI agent hits a case that is ambiguous, high-stakes, or outside of its scope. The question is whether it knows it should stop.

An agent can navigate interfaces and take actions on a user’s behalf. When encountering ambiguity, lacking context, or facing a decision they shouldn’t make, the agents are more likely to make an aggressive error if the system has no good way to say, “I don’t know,” or, more importantly, “I don’t have the authority for this.”

That’s the actual problem. We give agents buttons to click, forms to fill, files to move, money to send, and content to delete. But without a real abstain option, they will just keep going.

OpenAI’s research on hallucinations offers a useful reference point here: When systems are rewarded for guessing instead of acknowledging uncertainty, they guess. An agent with no real handoff or refusal state can do the same thing in a more dangerous form.

Authority must be part of the task

​​The cleanest way to think about this is to consider authority part of the task. If the workflow becomes ambiguous, high-stakes, or outside the agent’s scope, and the next action is the permanent deletion of files or content, or a large financial transaction, the action should probably be escalated to a human.

Many computer-using agent systems are designed around that logic. They ask for user confirmation before actions that have external side effects, such as submitting an order or sending an email, and they decline higher-risk tasks, such as banking transactions.

An incorrect answer from a chatbot is bad, but a wrong action in a live system is worse.

Treat abstinence like a success state

Organizations should define roles and responsibilities for human-AI systems and set criteria for what the system should refuse, according to the framework proposed by the U.S. Department of Commerce’s National Institute of Standards and Technology (NIST). 

The human-AI configuration itself is a risk area, including overreliance and automation bias, so transparent acceptable-use policies for chatbots and decision-making tasks can specify what the system should not do. In practice, that can mean drawing a hard line around sensitive data, high-consequence decisions, or actions that are outside the system’s role.

That framing is right, but I’d push it one step further. The “abstain state” should be treated like a success state for the right class of problems. When consequences go up, and confidence goes down, the system should stop and hand off.

You can see the failure mode in language models when they work on math problems. Instead of saying, “I don’t know,” they hallucinate. Computer-use agents can do the same thing in a more dangerous form. Without a real fallback, they keep moving through the workflow because continuing is the only behavior available.

That’s why I think execute, escalate, and refuse are the right three states. If those states are not built into the workflow, the system does not have a real way to stop.

Design the handoff before you deploy

Once you accept that abstain and escalate are core capabilities, the next question is operational: What does the handoff actually look like?

In practice, it doesn’t have to be fancy. It can be a push notification or an email. What matters is that the boundary is clear, the next owner is obvious, and the human has enough context to take over.

Anthropic has made a similar design choice with Claude Code, which requires approval before running commands or modifying files. It also notes that too much approval creates approval fatigue. Many approvals become routine for users. The ideal outcome is approval at the right boundary.

If the agent reaches a point outside its authority, it should be able to stop and say so. If the handoff is vague, the human is left figuring out what happened and what they are supposed to do next.

The workflow has to be specified clearly. Someone has to decide which actions belong in the execute bucket, which belong in the escalate bucket, and which belong in the refuse bucket.

3 boundaries to guide AI agents

Before you deploy an AI agent system as described, decide three things in advance:

1. What the agent can do on its own: Low-consequence, reversible work is usually the right place to start. That can include navigating an interface, filling out forms, moving information between systems, or teeing up an action for review, as long as the task is clearly inside the agent’s scope and easy to unwind if something goes wrong.

2. What must escalate to a human: Permanent deletion of files or content, large financial transactions, or anything with real external consequences should trigger a handoff, ideally through something simple like a push notification or email with enough context for the human to take over.

3. What it should refuse entirely: If the request involves sensitive data, a high-consequence decision, or an action the system is not authorized to take, the right behavior is refusal. The same is true when the system would have to guess, improvise, or act outside its role to complete the task.

If that boundary isn’t explicit, the agent will guess. That’s not what you want when the system can click buttons, move money, change records, and delete things.

Put the boundary in the workflow. Make the handoff explicit. Let the agent stop when it should. If the system can’t say, “I don’t have the authority for this,” it isn’t ready.

Neel Somani is the founder of Eclipse.