The future is now. Or in the case of AI development, the future is five minutes ago and the now is already outdated.
We are at the height of the hype cycle in the world of AI and ML. The rate of innovation is at a breakneck pace. The largest technology players in the world are pouring untold resources into being first—first to deploy the latest headline-grabbing ML model, first to commercialize and first to dominate the market and build a moat around the market share. It’s not strange to see the equivalent of modern-day gold rushes in the tech world, but when it comes to AI, it seems the pace of innovation and hype cycle that surrounds it is at a fever pitch.
REMARKABLE TECH STILL HAS LIMITATIONS
For example, take a look at Meta’s Segment Anything Model (SAM)—one of the more celebrated ML models to hit the market in an already crowded year. For those unfamiliar, SAM was introduced by Meta to great fanfare (at least in the Semsegmentation space) in April. In short, SAM was developed to help AI teams automate their image and video annotation processes by providing a powerful model trained on a very large dataset.
SAM’s promise is that with the world’s largest segmentation dataset of more than a billion masks and 11 million images, it is a powerful tool to generalize to new types of images and video. The technology is a remarkable feat in computer vision and has the potential to help AI teams prep and annotate their data faster than ever before.
But despite the impressive technology—and SAM is impressive—it’s important to remember that there are limitations. This is especially true when it comes to complex real-world use cases such as medical AI, where precision can be a literal case of life and death. Some say that with models such as SAM, the need for human-driven data annotation would be a thing of the past.
A SIMPLE TEST
Our engineers recently developed a test to put SAM through its paces. They used a complicated use case with many edge cases (those pesky anomalies that make AI models behave in unpredictable or inaccurate ways). In this case, they used publicly-available data from the National Institute of Health and tasked SAM with identifying tumors in radiology scans of lung tissue.
The test was simple: Test SAM’s capabilities alone, then compare SAM’s results with a model built by us using human experts. The test was designed to see how well each one met benchmarks. It should come as no surprise to anyone reading this far that the human expert model was far superior. SAM on its own was able to identify lung tissue in the scans, but it struggled to identify tumors.
The expert-trained model, however, was able to shine. In short, our human expert model was able to identify tumors at as much as a 95% accuracy rate. This number is significant, especially because in some parts of the world, radiologists have to deal with such a high volume of scans that their error rate can be as high as 70%. A tool that delivers even 75% accuracy can help drive that error rate down significantly, and save lives in the process.
The point isn’t that human expertise and training beat out SAM’s already extraordinary capabilities. That should be expected, especially when examining complicated data like radiology images. The point is that while the level of automation that is available to AI and ML pros is impressive, the hype cycle hasn’t caught up to the realities of commercialized AI.
As companies build purpose-driven applications that solve specific problems, the large models won’t stack up against the tuned models for that specific data flow.
PRECISION REQUIRES HUMAN EXPERTISE
This is where experts in the loop will continue to be important. Precision—especially when driven by the ML model du jour—requires human expertise. Many of the top AI, data and ML practitioners agree. In our recent State of MLOps report, 86% of the AI pros surveyed indicated that human labeling is essential, and as such, they are using expert-in-the-loop training at scale within existing projects. Additionally, 65% of respondents also stated that a dedicated workforce with domain expertise was required for successful AI-ready data.
Those proclaiming the demise of human-driven annotation in favor of strictly automated flavor may have been a little too quick on the draw. Although automation is impressive, it still needs people to be useful for AI models that address complexity or require precision. And I believe that is likely to remain true for the foreseeable future.
As much as things change in the world of AI, one fact remains the same: If you want to develop the next game-changing AI product, make sure you’re keeping human experts a key component in your development cycle.
Jeff Mills is the Chief Revenue Officer at iMerit.