Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

If you can’t debug your AI, you don’t control it

Leaders should ask their teams if they can diagnose and fix AI tools.

[Photo: Getty Images]

Profile photo of Neel Somani

Founder · Eclipse Labs

I keep running into the same problem when people talk about AI safety, interpretability, or alignment. Everyone argues about methods, and almost nobody agrees on the end state.

You see it everywhere. One camp of people wants explanations that feel intuitive. Another wants deeper mechanisms that generalize across model changes. A third group does not really care if the system works and gets better. All sound reasonable in isolation. None of them answers the question that matters:

What does success look like?

If you strip away the technicalities, this is a leadership problem. It is the same mistake companies make when they invest in dashboards that look good, research that sounds smart, or systems that optimize the wrong metric. You end up very busy, very sophisticated, and still unable to fix what broke.

Large language models are engineered software artifacts. If you treat them as something to be interpreted, you miss the fact that the teams building and deploying them can intervene on them, modify them, and constrain them.

That is why I think the right framing is debuggability—the ability and ease with which developers can find, diagnose, and correct defects. In practice, debugging breaks down into three steps.

1. Localization is the difference between stories and control

In any serious system, debugging has a shape. First, localize the problem. Then intervene. Finally, verify that the fix worked and did not break something else. If you cannot do all three, you are just guessing.

Localization means identifying which internal mechanism is responsible for a behavior.

In the strongest form, this means you can ask counterfactual questions, like:

  • Can the behavior occur without this mechanism? 
  • Can the mechanism activate without the behavior?

If the answer is yes, you should be able to surface concrete examples. If you cannot, you do not really know where the problem lies.

This eliminates explanations that fail the moment you intervene. An explanation that collapses under intervention is not an explanation.

2. Intervention turns understanding into engineering

Intervention is where things get uncomfortable. Once you claim a specific internal mechanism is responsible for producing a behavior, you should be willing to touch it. You should be able to modify a head, a multilayer perceptron (MLP), or a subspace in a targeted way.

The intervention should remove the undesired behavior in a specified domain without creating collateral damage. If changing the mechanism causes unpredictable side effects, then your localization was not as precise as you thought.

This is the moment where interpretability stops being an academic exercise and starts looking like engineering.

3. Certification forces you to be honest about scope

The final step is certification. This is where most people quietly give up, because it forces you to be explicit about scope. You are not proving that a frontier model will never do anything bad. That is a category error. You are making falsifiable claims about what a system can and cannot do within a clearly defined boundary.

For a formally specified input space, you should be able to say things like:

  • “This subcircuit cannot activate a forbidden feature.”
  • “This intervention preserves all other behaviors in scope.”
  • “This pathway cannot bypass a guard unless a specific internal condition is met.”

These are universal claims. They are not probabilistic and do not rely on sampling.

This is also why formal methods are not optional. If you want to make claims about impossibility, preservation, or closure under intervention, you need machinery to express and check them.

These tools exist to let you say, with a straight face, that something cannot happen within a defined boundary.

The analogy I keep coming back to is large, safety-critical software. Nobody can prove that a browser will never crash. What they prove is that specific routines are memory safe, that sandboxing enforces certain invariants, and that a patch eliminates a vulnerability without regressions. The guarantees are partial, local, and compositional. They are exact as to where they apply.

That is the right mental model for AI systems.

DEBUGGABILITY IS THE ONLY STABLE END STATE

Debuggability does not always mean reverse-engineering a model into a single, clean set of rules. Systems do not always work that way. They rely on many overlapping mechanisms that act simultaneously, and any useful abstraction must preserve that flexibility rather than erase it.

Debuggability also does not mean identifying a single, privileged cause of an outcome. Behavior flows through deep, branching pathways with redundancy and overlap. What matters is whether the mechanisms you identify are sufficient to explain and control the behavior within a defined scope.

If there is a bypass, you should be able to find it by actively testing the system.

For leaders, the takeaway is simple: Do not ask whether a system is interpretable in some abstract sense.

Ask whether it is debuggable. Can your team localize failures? Can they intervene surgically? Can they certify what changed and what did not?

If the answer is no, you are flying blind, no matter how impressive the explanations look.

The endgame here is leverage. Debuggable systems let you move fast without lying to yourself about risk. They let you make bounded guarantees, and they let you treat AI the way it exists in the world as a system you are responsible for maintaining.

Neel Somani is the founder of Eclipse Labs.