In the corporate arms race for AI supremacy, most leaders are asking the wrong question. They are obsessing over which “brain” to buy, debating GPT-5.5 versus Claude Opus 4.7 versus Gemini 3.1 as if the engine alone wins the race.
The model is becoming a commodity. If your entire AI strategy is built on the back of a third-party API, you haven’t built a competitive advantage; you’ve just rented a utility. The real race is elsewhere, as the most sophisticated AI deployments today are constellations of purpose-built models built to solve specific business problems.
True differentiation is possible from orchestrating a proprietary system. In my previous column, I made the case for building specialized, domain-trained models. Once you have them, you need to determine how they work together.
FROM THE ASSEMBLY LINE TO THE OPEN OFFICE
Enterprise AI systems used to work like factory floors. Model A passes its output to Model B, which hands it to Model C. Linear, predictable, easy to audit. A lot of production systems still run like an assembly line, and for the right tasks, it’s fine.
The field is moving toward agentic orchestration, where models talk to each other, flag what’s missing, and go find it without waiting on a fixed chain of instructions. But most production systems, including ours at Dataminr, take a hybrid approach. We use an assembly-line model where we need it for speed, efficiency, and cost. We branch out into a constellation of models at specific points where complexity demands it. Multiple models operate in parallel on the same input, or hand off to a human reviewer when the task warrants it.
WHAT THE ARCHITECTURE GIVES YOU
The business case comes down to control and cost. Control is about flexibility: routing, adjusting, and improving the system on your own terms. Cost is about scale. Running complex workflows through a single general-purpose model at enterprise scale gets expensive. A system built around specialized components is higher quality, easier to audit, and costs significantly less once it’s in production, especially if you are using smaller models. For high-volume tasks like our daily alert processing, using external flagship APIs can cost over 100 times more than running purpose-built in-house models. For one part of our pipeline alone, that’s the difference between $20 and $2,500 per day.
Each specialized component also outperforms what a general-purpose model would do in that role. Cost goes down; quality goes up.
The system is decomposable. Each model owns a specific part of the pipeline. If it underperforms, you fix it without touching anything else. Our policy is to take the best approach that meets our accuracy, cost, and latency requirements, whether in-house or, in limited cases, external. Owning the architecture gives you that choice.
WHAT CHANGES FOR YOUR TEAM
Building these systems changes how teams operate, and the role of humans-in-the-loop is where that change is most visible and most often misunderstood.
In the traditional model, humans label data to train models, run quality checks to catch drift, and review outputs before anything goes out the door. Typically, models handle routine work—pre-labeling data, assigning categories, drafting identifying information—while a human reviews, edits, and approves. This is structured and sequential, along with being predictable. It is the factory floor, again.
That model hasn’t disappeared. But today working with external LLMs reduces your need for labeled training data. Evaluation can increasingly be handled by a second model—an LLM-as-judge—enabling more frequent and comprehensive checks than a human team could run. Someone still has to calibrate the judge. The human role shifts from performing the evaluation to ensuring the evaluator is trustworthy. That’s a different skill set.
The barrier to building has also dropped. Non-technical staff can now prompt models, assemble agentic workflows, and deploy without writing a line of code. Your AI team is still building models, but also building infrastructure for subject matter experts to deploy effectively and responsibly. Quality control and creation have merged into the same role.
Models can now interact with datasets, call APIs, and coordinate with each other, with or without supervision. Our vision at Dataminr is to have humans orchestrate agents rather than review individual outputs. The human role becomes more about directing the flow, versus catching errors.
Humans remain important, with the placement of human expertise and judgment becoming more important. We start asking what humans should own, rather than where to put a human in the loop.
WHAT IT TAKES TO GET THIS RIGHT
A few things can trip up teams building these systems.
- Choose your task carefully. Not every problem needs this kind of architecture. Some tasks are narrow enough for a single well-tuned model to handle; adding orchestration introduces cost and complexity with no benefit. If you find yourself needing two or three models, that’s when it makes sense.
- Build in evaluation from the start, at the component level. A system that translates, summarizes, and organizes across domains needs a separate rubric for each piece. Evaluate components separately, then end-to-end. That takes more time. When something underperforms, you know exactly where to look.
- Design for latency and failure. Multi-component systems have more places to stall. A slow model can block everything downstream. Timeouts and monitoring aren’t glamorous, but they separate a system that works in production from one that’s demo-worthy.
- Decide what humans own. As the system becomes more autonomous, the temptation is to reduce human involvement. The better move is to figure out where to concentrate it. Human judgment moves upstream, toward decisions about architecture, evaluation, and direction. Design for that intentionally, or it won’t happen.
The frontier models will keep getting better. So will everyone else’s. What they can’t replicate is your architecture, your data, and the human judgment woven throughout. That is the competitive advantage worth building.
Joel Tetreault is chief AI officer of Dataminr.