There’s a good chance your company is stuck in purgatory. Not the purgatory of Dante’s Divine Comedy from the 14th century — a bad data purgatory that so many companies find themselves stranded in, unable to make meaningful progress toward their AI goals.
DATA QUALITY: THE BOTTLENECK TO MANY AI INITIATIVES
The data purgatory hits major technology initiatives like AI projects when obstacles are created by data accuracy, quality, and accessibility. Fast Company has put a spotlight on the make-or-break impact of data quality on AI initiatives in a number of recent articles. I have also discussed this in my recent Fast Company Executive Board column.
Data can suffer from many quality issues. Is it accurate? Is it precise enough and rich enough for how AI will use it? Is it accessible when you need it? Is it current? Is it in the right format? Is it properly structured? Is it compliant with evolving regulatory requirements like privacy laws?
These issues arise particularly with technical data, where accuracy, precision, timeliness, and other characteristics are crucial for an organization’s operations. Examples include geospatial/GIS data, infrastructure operations data, manufacturing/industrial systems data, IoT implementations data, and more, which are critical across a wide range of industries.
For organizations that rely on this complex technical data, ensuring that the data is truly AI-ready can be challenging. The traditional remedies tend to be slow and costly because they rely on manual review of data sets and manual fixes to make the data points more accurate, remove errors, enrich the data, improve dataflows that fix latency issues, and so forth.
Given how vital data is to every aspect of an organization’s operations, investing in greater data quality pays dividends. But how can your organization accomplish this in a way that is cost- and resource-effective, while also moving at the speed needed to achieve aggressive AI initiative timelines?
COULD AI ITSELF BE THE SOLUTION?
My team offers a surprising answer to that question: Making data truly AI-ready may require the use of AI.
To illustrate this, let’s look at a municipality in Illinois that is working with TRC to deploy AI to drive operational efficiencies related to infrastructure management and delivery of public services. As is true of many organizations, an audit of its data revealed data quality issues that would stand in the way of successfully building and training the AI models. And like most organizations, this client has limited resources to address these data issues with the time-consuming techniques I discussed above.
But limitations drive creativity and necessity drives invention. That is how we began experimenting with using AI to automate the process of identifying and resolving data quality problems. Based on an assessment of the quality issues we were seeing and anticipating in the municipality’s data, we trained a generative AI model to review massive datasets for missing data points, outlier data points that might be errors, data latency issues that would require improved dataflows, and more. We also trained the AI model to assess data quality and precision in the most complex datasets, allowing the model to assess data with some of the same expertise and nuance that a trained professional would use in reviewing the data.
After reviewing the assessments, the AI model was trained to perform fixes for common quality issues and escalate others for manual resolution. These automated steps were overseen by professionals, acting as a force multiplier that allowed a small team to perform data enhancement to massive datasets that would have been impossible otherwise.
Nothing can replace the data enhancement work that a trained professional with domain expertise can perform, but these pilot projects show that AI has an important role to play in augmenting teams who face urgent timelines for ensuring that data is AI-ready. I have a career’s worth of working on data enhancement projects, and I was pleasantly surprised at how effective this approach was. My team has used a similar approach with other projects, and the outcomes are equally positive. The result is a process that could prove to be a valuable blueprint for other organizations trying to make it through the data purgatory and get their AI initiatives back on track.
MOVING FORWARD WITH DATA
It is important that organizations understand what data quality means to them and have a data governance program that meets its own set standards for quality. Begin with a consensus about how to define data quality. This foundational definition will give your organization a clear standard for diagnosing and resolving issues in order to be AI-ready. My next column will cover this topic in depth.
Until then, if you get stuck in the data purgatory on the way to achieving your AI objectives, don’t worry. AI may be exactly what you need… to take full advantage of AI.