Years ago, as a marine engineer with Seaspan Ship Management, I was sailing on a 5,000 Twenty-foot Equivalent Unit container ship transiting the Panama Canal on the Asia-to-USA trade route. Looking out of the porthole one morning, I rubbed my eyes and ran to the deck. From the rail of a vessel four city blocks long, I could see no open water on either side. It felt as though we had driven an ocean liner onto land.
Our ship’s width measured 32 meters. The Panama Canal’s navigable channel is only a half meter wider at 32.5 meters. That is no accident. It was a deliberate economic calculation by the shipowner and understanding it changed how I think about the supply chain.
A container ship does not spend most of its life in the canal. Most voyages run port-to-port on routes that never touch it. Building a vessel to the maximum width of the canal gives the operator the largest, highest-capacity ship for every ordinary voyage, while preserving the ability to transit Panama whenever the route calls for it.
But it comes with one consequence nobody on board is ever surprised by.
The consequence of crossing the canal
The paint on both sides of the hull will come off during transit because the ship contacts the canal walls. Everyone on board knows this before entering, and the deck department has the touch-up supplies ready before exiting.
When this happened during my crossing, we did not hold a root cause analysis meeting. We did not form a committee to investigate why the paint had abraded. We did not file incident reports or brief leadership.
The deck crew picked up the brushes and went to work.
The energy that might have been spent on analysis and blame was redirected into the only thing that mattered: getting the ship ready for the next leg. The damage was not a failure. It was a cost of the strategy, fully anticipated and already budgeted for.
Some failures are anticipated
I have spent 15 years running manufacturing and supply chain networks at the national level, managing thousands of truckloads a day across plants, third-party logistics partners, and distribution nodes. The most important philosophy I carried from my years at sea is this: In a truly optimized system, some things are going to break by design.
Not by negligence. Not by poor planning. By design.
When you push utilization, transit schedules, carrier commitments, and inventory to their efficient frontier, the system will experience disruption. Some disruptions are the price of that extraction, and the cost of preventing them is higher than the cost of absorbing them.
The most damaging action a supply chain leader can take is to treat a known, manageable failure as a crisis that demands investigation. When a carrier misses a window, a warehouse runs short on a SKU, or a lane goes down for 48 hours and your model anticipated these as probable, the worst response is to redirect your team into root cause analysis.
You will waste the one resource you cannot recover—your team’s bandwidth and morale—to solve a problem that was never meant to be solved, only absorbed.
3 failure types to understand
Operational leaders must differentiate among three failure modes.
1. Known, recoverable, and budgeted
These are the paint-off-the-hull moments; the response protocol is predetermined and execution is the only variable. Move immediately to repair.
2. Known, recoverable, but not budgeted
Something your model said was low probability went wrong. Fix it first, then investigate, but keep the investigation proportionate. The goal is model recalibration, not organizational catharsis.
3. Unknown and potentially structural
Something happened that your model did not anticipate, and it may indicate a flaw in how you mapped the system. These failures demand full investigative rigor. This is where you spend root cause analysis capital.
Most teams thrive in firefighting mode and treat every failure as unknown, pushed there by escalations from sales, commercial, or finance. The result is a team perpetually in investigation mode, demoralized by disruptions that were never meant to be prevented, and unable to see when a genuinely structural problem emerges from the noise.
The morale cost no one measures
When a team is repeatedly asked to explain events that the network’s own economics made inevitable, people begin to feel that no level of effort is sufficient, that leadership does not understand how the network works, and that their jobs are about managing blame rather than logistics.
On that ship transiting the Panama there was no demoralization when the paint came off. There was competence. There was readiness. There was a crew that trusted leadership had thought through the strategy honestly. That trust is the most valuable operational asset a supply chain leader can build, and the most fragile.
How to fix the problem
For every major failure your network has experienced in the last two years, ask your senior operations team a single question: Was this a designed-in outcome of an anticipated optimization, or a genuine deviation from the model?
For the designed-in outcomes, build response protocols that eliminate the investigation step and redirect that energy into recovery. Then, and this is the part most leaders skip, communicate it to your team. Tell them what is expected to break, why, and what the response plan is, and watch the morale shift.
For the genuine failures, invest fully in understanding them. That is where the real lessons about your system live.
Final thoughts
The Panama Canal does not reward caution. It rewards operators who understand they are optimizing for a half meter of clearance on each side, that the cost is the paint on the hull, and that the best response is to have a crew with brushes ready.
The supply chain leaders who build the most resilient operations are not the ones who eliminate disruption. They are the ones who know exactly which disruptions they have built the muscle to absorb without missing a step.
Niraj Jha is an operations executive and the author of From Engines to Algorithms.