Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

The hidden costs of AI models

Many organizations find themselves in an accelerated “AI arms race," but few organizations have the resources to develop and deploy a model from scratch.

[Images: Lucky Ai/Adobe Stock]

Profile photo of Kunal Agarwal

BY Kunal Agarwal

When ChatGPT exploded onto the scene, it captured our imagination in a way that no other AI had before. Seemingly overnight, AI is all anyone can talk about. Many organizations find themselves in an accelerated “AI arms race” to avoid falling behind their competition. More than ever, AI has become a strategic imperative.

AI capabilities, including large language models (LLMs) and other generative AI, are no longer just theoretical data science projects. Innovative modern cloud data platforms—Databricks, Snowflake, BigQuery, Amazon EMR—have helped put practical production-grade AI/ML into the hands of more users from a wider variety of departments in every company. But all this increased utilization puts a lot of financial pressure on the business, from the C-suite to finance to the lines of business to engineering team leaders.

Because there’s no getting around it: AI models are extremely complex and massive data workloads, and that makes them expensive. How expensive? A recent McKinsey report estimates the total cost of ownership (TCO) for three different LLM archetypes:

  • Taker—using off-the-shelf models, with little or no customization: $2 million
  • Shaper—customizing models to integrate with internal data and systems: $10 million
  • Maker—building and training your own foundational model: $200 million

Few organizations have the resources to develop and deploy a foundational model from scratch, but even some form of a taker/shaper model costs several millions of dollars to develop and deploy. Most enterprises are already struggling with escalating cloud data costs, so business leaders are caught in a crossfire between increased demand for ever more of these petabyte-scale data workloads and the financial reality of budgetary (and staff) constraints.

This is why data-forward enterprises are increasingly adopting a FinOps approach to AI models, bringing finance, engineering, and the lines of business together to collaboratively ensure that the organization is realizing the greatest value from its cloud data investments. But that requires knowing the true costs of these initiatives, and a lot of these costs are “hidden.” We’ve seen that companies that mitigate these hidden costs find that they already have enough resources—human and financial—to run as much as 50% more AI workloads. 

But first, you have to understand where the money is going. 

MEASURE UNIT COSTS

The lion’s share of the cost for AI models goes to building and running the data pipeline(s)—say, on Databricks or Snowflake or BigQuery—to train the models. For a highly complex AI model pipeline, that’s probably 10,000s of individual jobs. And each job carries a price tag. 

You need visibility into costs at a granular level: how much each individual job costs, how much each data engineer/data scientist working on the project is spending, how much the different applications they’re using cost, what the data storage costs are, etc. 

Without visibility into unit costs at the application and/or user level, you can’t tackle the biggest hidden cost—data pipeline inefficiency.

THE HIDDEN COSTS OF DATA PIPELINE INEFFICIENCY

The single biggest hidden cost of AI lies in inefficient data pipelines. Your monthly cloud data bill contains a tremendous amount of self-inflicted overspending that adds up to millions of dollars of “waste” over the course of a year. Eliminating this overspending goes a long way to funding additional AI projects without increasing budget.  

The problem is, the more users leveraging AI models, the more inefficiencies (and unnecessary costs) creep into the data pipelines. While having more departments run AI pipelines is a good thing—it means that the AI is delivering business value—the people running these workloads simply do not have the necessary expertise with cloud data platforms to know how to run data pipelines efficiently. 

When most people think of unnecessary spending waste in their monthly bill, they think of infrastructure. And indeed you can free up a lot of budget (cloud data platform owners tell me 30% or more) by shutting down idle clusters and right-sizing infrastructure: configuring the number, size, and type of resources needed to run the job—no more, no less. But a more insidious culprit of overspending lies in the code itself. Inefficient code in production causes pipelines to run longer than they should or even fail outright (and so cause these expensive pipelines to be run again and again).

More companies are turning to AI to root out such inefficiencies. Given the complexity and high stakes of AI pipelines, with all their business-critical interdependencies, data engineers need some kind of automated intelligence to collect and correlate the hundreds or thousands of details from logs, traces, metrics, events, and other metadata from dozens of different components within the entire data stack and then make sense of it all in order to optimize their jobs. It’s become more than humans can handle by themselves—especially at enterprise scale—at the speed demanded by the business.

HIDDEN OPPORTUNITY COST OF MISDIRECTION AND PRIORITIZATION 

Not every AI model pans out. The 2023 State of Data + AI Report finds that only one out of three AI models ever make it from pilot to production. But it still costs millions to run an experimental AI model. Before you commit to this kind of expenditure, be sure you have a solid business use case for the AI model. All too often, organizations get caught up in the hype and excitement of the “New New Thing” (like LLMs), which doesn’t yet have proven business value. 

There’s also the opportunity cost of how your scarce data engineering experts spend their time. Complex AI pipelines need troubleshooting—usually by your top engineers. Keep a watchful eye that you don’t devote precious data engineering resources to novel projects without proven business value at the expense of business-critical AI models and data analytics pipelines.

AI has transformed the competitive landscape of every industry. But AI requires a significant investment of resources—both money and people—and its hidden costs can derail an organization’s ability to compete. Uncovering and correcting these hidden costs is a key differentiator that will separate the winners from the also-rans.


Kunal Agarwal is the CEO of Unravel Data, specializing in data observability and FinOps for the modern data stack.