For the last two years, most conversations around enterprise artificial intelligence have centered on which models to use, how many graphic processing units are needed, and how to get employees to adopt the technology. Those were the right conversations during enterprise AI’s nascent phase.
As AI moves from answering simple questions to operating as agentic systems that handle complex, multi-step tasks, the conversation now must focus on how to run AI economically. And one of the most expensive parts of that equation is something many executives have never heard of: the recompute tax.
RECOMPUTE TAX BASICS
The recompute tax is the cost paid when an AI system is forced to regenerate context it has already processed because prior work was not retained or efficiently retrieved during inference.
Imagine asking a senior analyst to write a board memo, but every time they finish a paragraph, you take away their notes. Before they can continue, they need to reread the source material, rebuild the outline, and reconstruct the logic they had already developed.
Many AI systems face a version of that problem during inference. In a multi-step task, the model builds a working context as it reads information, makes decisions, and moves from one step to the next. But because GPU memory is limited, context is not always retained or cannot be retrieved fast enough. As a result, the system needs to rebuild the context using expensive GPU cycles.
The system’s repeated work is the recompute tax. It shows up as lower effective GPU utilization, higher infrastructure costs, and more energy required to produce the same result.
CONTEXT AS A NEW COST CENTER
Most executives understand the cost of training a model. But inference—the act of using the model in production—is where AI becomes a recurring operating cost. A simple prompt might only cost a few cents to execute. As that interaction becomes a complex, multi-turn workflow, the cost can rise to a dollar or more. Add the recompute tax, and the same task may cost twice as much because the system is rebuilding context. If you multiply that across thousands of requests a day, the costs can skyrocket.
Longer recomputation also means more GPU cycles, more power consumption, and more infrastructure dedicated to recreating work the system has already performed. At a time when data center power demand is already under scrutiny, paying to repeat the same computation becomes unsustainable.
The memory requirement also compounds quickly. A long-context conversation can require tens of gigabytes of active context memory. In fact, a single 128K-token conversation consumes about 40 gigabytes of context memory. In other words, an AI conversation equivalent to about 300 or 400 pages of text would require more than twice the working memory of a typical laptop for a single AI interaction.
These are the symptoms of today’s AI architectural fail points. The more complex the question asked of an AI agent, the more expensive it becomes.
REPETITION DOESN’T MEAN PRODUCTIVE CAPACITY
Not all GPU utilization is valuable utilization.
A GPU can consume power, occupy capacity, and contribute to latency while generating no new value. For example, when context is evicted or unavailable, the model cannot continue where it left off. Instead, it needs to rebuild the context, causing the GPU to recreate the work it has already performed.
Leaders need to look beyond utilization metrics alone. The better questions to ask are:
- How much of our inference capacity is producing new tokens versus rebuilding old context?
- How often do our systems lose useful state between requests, sessions, users, or agents?
- Do our AI economics improve with scale, or do they degrade?
The more useful metrics we find are time to first token, time per output token, tail latency, power draw, memory efficiency, jitter, concurrency, and the ability to reuse prior computation. These metrics determine whether an AI system feels responsive to users and whether it makes financial sense to operate.
THREE QUESTIONS TO GUIDE AI MEMORY STRATEGY
C-suite executives today need an AI memory strategy. I’m not saying that every CEO needs to understand every detail of the GPU memory, remote direct memory access, or cache management. Still, leadership teams should understand where AI systems store working context, how long that context persists, who can reuse it, and what happens when it is lost.
There are three questions every leadership team should ask to develop an AI memory strategy.
First: What is our current agentic AI usage, and what is our strategy? Leaders need to know their agentic AI usage, how many workflows will require persistent context across steps, and expected growth.
Second: What is our cost per successful task, not just cost per token? A cheap token is not cheap if the system needs to regenerate thousands of prior tokens to complete the work.
Third: Are we optimizing the full AI data path? AI infrastructure is no longer just storage below and GPUs above. There is an emerging middle layer where active context, memory, and inference state live.
Miss any one of these and the system starts paying the recompute tax again.
THE NEXT AI ADVANTAGE: ARCHITECTURAL
Companies that succeed in the next AI adoption phase won’t just train better models. They will know how to retain and reuse the necessary intelligence for each task the AI is given. They will shift AI use from recomputation to retrieval—from isolated runs to continuous ones. They will nail the necessary metrics, such as better GPU utilization and lower cost per inference, and make real-time, multi-step AI viable.
To meet these parameters, leaders need to rethink infrastructure. Systems built for training, storage, or isolated model runs were not designed for long-running, context-heavy workloads. Better GPU utilization, lower cost per inference, lower latency at scale, and more reliable multi-step workflows are the foundation for making AI economically viable in production.
AB Periasamy is founder and CEO of MinIO.