One of the biggest mistakes I see businesses make is choosing AI models based on benchmark charts they don’t understand.
Every model release follows the same pattern. The AI lab publishes a leaderboard, declares that its new model is the best, and includes a collection of graphs showing improvements over the competition. Most people accept those numbers at face value without asking a much more important question: What is this benchmark measuring?
Every AI lab has an incentive to present its model in the best possible light. They’ll highlight the benchmarks where they perform well and ignore the ones where they don’t. Sometimes they’ll even present graphs with skewed axes to make relatively small improvements look much bigger than they really are.
RELY ON INDEPENDENT BENCHMARKS
This is why I rarely rely on the benchmark charts published by the AI labs themselves. Instead, I use a third-party site like Artificial Analysis, which aggregates results across models from every major provider. It gives you a much more balanced picture and lets you compare performance across many categories instead of whichever ones happened to make the latest release look impressive.
The next thing you’ll notice is that an enormous number of benchmarks are focused on coding. That’s not because AI companies think everyone is a software engineer.
There are two reasons. First, coding is easy to verify. A program either works or it doesn’t, making it one of the easiest tasks to evaluate objectively at scale.
Second, coding ability transfers surprisingly well to many tasks that don’t look like programming. These tasks include organizing folders, editing thousands of files, transforming spreadsheets, extracting information from documents, and automating repetitive business processes. Many of these workflows are powered by Python, an open-source programming language, or similar tools behind the scenes. Strong coding models tend to perform well on these tasks because they’re solving the same underlying problems.
CHOOSE BENCHMARKS THAT MATCH YOUR INDUSTRY
If your business operates in a specialized field, resist the temptation to only look at general leaderboards. There are benchmarks specifically designed for finance, biology, medicine, law, mathematics, and scientific research. Those will usually tell you much more about how a model will perform in your day-to-day work than a general benchmark ever will.
For broader capability, benchmarks like OSWorld are worth paying attention to, as they measure how well models complete realistic computer tasks. Just make sure you’re looking at the latest version. AI models improve so quickly that many older benchmarks have become too easy, forcing researchers to release v2 and v3 versions that are much more challenging.
Where frontier models, the most advanced AI systems, still struggle is maintaining context over very long tasks. If a project takes more than an hour to complete, you’ll often see the model lose track of earlier decisions, repeat work, or drift away from the original objective. This is improving quickly, but it’s still one of the biggest limitations today.
Finally, don’t just read the score. Read the benchmark itself. Most major benchmarks are completely open source, including test cases and evaluation methodology. Spend a few minutes looking at the tasks and ask yourself whether they resemble the work you expect the model to do. That will tell you far more than any leaderboard ever will, and then you’ll be able to determine which AI model is best for your business.
Komninos Chatzipapas is the founder of Omicron AI Software.