Most of the AI safety conversations I hear in boardrooms start from the same assumption: The model is the problem. Improve the model, evaluate it harder, add more guardrails, and at some point it becomes safe enough to hand real work to do. I don’t think that is the right objective, and I suspect some of the people saying it don’t entirely believe it either.
Models will get more capable. They will also stay probabilistic and adaptive, and they will keep doing things nobody designed them to do. That isn’t a bug that gets fixed in the next release. It is what these systems are. And the more useful they become, the more that uncertainty costs when it goes wrong.
So the question for an institution isn’t how to make AI safe. It is how much authority you are willing to give it, and whether you can keep hold of that authority once you have.
DETERMINE THE SCOPE
The difference is easy to miss because the technology looks the same from the outside. A chatbot can draft an email. But an agent can read the customer record, send the email, amend the account, and release a payment, even if the model underneath is identical. What has changed is the combination of what the agent can do, what it can reach, and how much of it happens without anyone looking.
That is also why security teams, who are usually the first people asked to own this, find it slips through their fingers. Security exists to stop unauthorized behavior, the stolen credential, the compromised endpoint, or a bad actor. An agent can cause serious damage while fully authenticated. It can use the permissions it was granted, pursuing the set objective. Nothing has been breached. The agent did what it was allowed to do. The trouble is that what it was allowed to do was never properly decided.
I spent 30 years in markets, and one thing you learn is that you cannot test your way out of a problem whose state space you can’t enumerate. That is where agents are heading. Give one more tool and longer chains of actions, and the number of possible paths grows faster than any test suite can follow. Testing tells you a lot, but it cannot tell you that every future combination of model, data, tools, and environment will be benign. That’s because it hasn’t seen most of them.
BEHAVIORAL AND STRUCTURAL CONTROLS
This is why, in our own work on operational risk, we draw a hard line between behavioral controls and structural ones. A behavioral control asks the model to behave, such as putting “never delete customer data” in the system prompt. A structural control removes the delete tool from what the agent can call. I would not bet a company on the first. I would on the second.
Follow that logic and you end up with a fairly specific architecture. Every consequential AI system needs an explicit operating boundary, written down: what it may do, what information it may reach, which systems it may act on, how much autonomy it has, what needs a human signature, and what conditions should restrict or stop it outright.
And the boundary cannot be approved once and filed. Models get swapped. Prompts get edited. Data changes. Someone adds a connector on a Friday afternoon. Permissions widen because that was easier than narrowing them. An approval signed six months ago may describe a system that no longer exists. Companies need to question whether the system is still operating inside the authority they meant to give it.
I think this eventually becomes an economic discipline, not just a technical one. Boards don’t run credit risk or cyber risk with red, yellow, and green labels. They ask about exposure, expected loss, concentration, tail risk, and capital. AI will get there too, probably faster than people expect, because the alternative is either to keep agents on trivial work indefinitely or to keep discovering the boundaries after the fact.
The prize is real, and locking AI up is the wrong way to protect it. The better answer is to let it do more, and to put a boundary around that freedom which the machine itself does not control. That is how trust gets built with anything consequential. It won’t be different here.
Vinit Sahni is founder of governr.ai.