Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

Reliability is the hidden foundation of enterprise AI success

AI technology can evolve along with and beyond a company's needs, but reliability to critical for ensuring that progress can take place.

Reliability hidden foundation of enterprise AI success

[Adobe Stock / Grispb]

Profile photo of Allen Brokken,

AI Hypercomputer Specialist · Google Cloud

Most traditional enterprise systems are one-dimensional. They do what they’re designed to do—full stop. Once deployed, they either perform or they don’t, and when something breaks, you fix it.

AI-powered systems are different because they’re capable of evolution. Each training run is more than a simple technical iteration; it’s a moment of potential emergence. Behavior you didn’t explicitly build into your systems can surface as unexpected value, like when a model trained on text suddenly demonstrates multi-step reasoning. That leap is where innovation starts to compound. However, that emergence depends on whether the job completes. When a job fails, what’s lost isn’t limited to compute or time. The momentum derails, creating a missed opportunity for value to crystallize.

This is what I call “AI compound interest”—the accumulated return on uninterrupted iteration. But that value only builds if each step in the process completes and contributes to the next. Interrupt the chain, and you reset the curve. More than time, recovering that momentum can cost you strategic ground. That’s why reliability is more than a technical checkbox. It’s key to success.

AT SCALE, FAILURE IS A CONSTANT, NOT AN OUTLIER

Unfortunately, most leaders overlook the fact that reliability doesn’t scale like compute does. A single node with 99.9% uptime seems dependable, but multiply that across 1,000 nodes, and those small gaps add up fast. Teams can tune models, refine pipelines, and optimize every layer, only to lose hours of training to a silent node failure. No failover, no checkpoints—just progress, wiped. This is an operational reality that reveals a core design truth: AI success isn’t built on flawless execution. It depends on systems designed to take a hit and keep going.

I’ve seen this play out firsthand. In a past role, my team and I managed a dozen servers under a 99.9% SLA, which allowed for about eight hours of downtime per server, per year. We invested heavily in what we thought was best practice: premium disk controllers, mirrored drives, and 24/7 support contracts. It looked bulletproof.

However, when one of those premium controllers failed, it took 48 hours to replace. That single failure obliterated our SLA margin and consumed our operations budget. Meanwhile, the lower-tier components we’d expected to fail never did.

The problem wasn’t that we were underprepared. We just weren’t prepared for the right thing. We’d protected against low-risk threats and missed the higher-impact failure mode entirely. Reliability isn’t about covering every edge case. It’s about knowing which failures are most likely to cost you, then designing your recovery model around them.

NOT EVERY SLA IS BUILT THE SAME

Too often, infrastructure decisions are made on the strength of vendor marketing—99.9% uptime, multi-zone distribution, racks of GPU capacity. While those numbers sound concrete, I find that they hide the real-world operational behavior. That’s why I started asking, “What margin exists behind that promise?” For example, if a node fails mid-training, how quickly is it detected and how is the job reassigned? Do they checkpoint progress automatically, or does recovery start from zero?

One provider might expect you to build around failure by overprovisioning and absorbing the risk. Another may design failure recovery into the architecture itself, enabling graceful recovery without human intervention. Both may have the same SLA on paper, but they represent a different experience in practice.

TRAINING AND INFERENCE REQUIRE DIFFERENT PLAYBOOKS

Another common issue I see is treating training and inference as if they operate within the same reliability domain. Though they’re both parts of the AI lifecycle, their operational realities and failure consequences couldn’t be more different.

Training workloads are long running, highly distributed, and sensitive to synchronization breakdowns. One missed checkpoint can set a project back by hours, even days. Inference, on the other hand, plays out in milliseconds. It’s user facing, latency sensitive, and unforgiving. A hiccup in one microservice can instantly ripple out to thousands of users. So when teams apply the same reliability strategy to both, they often end up overcompensating in one area and underestimating risk in another.

THE SILENT LIMITER: POWER INFRASTRUCTURE

Despite the fact that it governs everything, power is a risk that most leaders don’t factor in. At hyperscale, power is a physical constraint. When you initiate a large-scale training run, the sudden spike in energy demand can overload the entire localized infrastructure. That could result in a specific rack of equipment shutting down or, in more extreme cases, an entire section of a data center hall.

I’ve experienced the challenge of diagnosing intermittent failures that present as issues with individual nodes, then turn out to be the result of too much power being drawn at once. That’s why I’ve started advising leaders to treat energy orchestration as a first-class reliability concern, on equal footing with compute and storage.

WHAT RELIABILITY REALLY MEANS

Too many teams treat reliability like an insurance policy. But in practice, it’s something deeper. If AI is going to deliver strategic value, reliability has to be part of the foundation. It’s what makes iteration sustainable and ensures the output of today’s training cycle becomes the input for tomorrow’s breakthrough.

It shows up across the board—in the speed at which you can retrain, the confidence your team has to push models into production, and the frequency with which those models ship without rollback. These signals reinforce the same truth: reliability is embedded in how consistently everything works when it matters most.

As a leader, it’s time to start asking your team, “Are we building for failure, or reacting to it? Are we compounding our innovation, or constantly clawing it back? When a failure occurs, do we lose models, or just milliseconds?” Because in AI, velocity may get you to the frontier, but only continuity lets you build on it.