Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

Why recent legislation falls short on consumer demands for data privacy

What should companies do when it comes to AI and maintaining and respecting data privacy?

Padlock over a keyboard

[Images: pla2na / Adobe Stock]

Profile photo of Tom Chavez

At the end of October, The Biden-Harris Administration unveiled its long-awaited executive order on AI. The scope of the directives laid out is impressively wide, but, from my perspective, we all should consider it more as a mission statement than a binding requirement leading to actionable outcomes of interest. I have called for companies to do better on privacy while regulation stays decades behind.

But here we are, left with a big, unanswered question: What should companies do when it comes to AI and maintaining and respecting data privacy?

First, don’t assume your consumers don’t care. Second, don’t try to accommodate all the existential threats when it comes to privacy because running an effective privacy strategy means staying focused on real-world, here-and-now consequences. 

Let’s look at the latter point for a moment: I’ve spent most of my career building companies that put data at their center. I still see privacy issues being delegated out to lawyers as a “compliance issue” and not as a strategic imperative for businesses seeking to use data and AI to drive growth. A new study commissioned by my team at The Ethical Tech Project and Ketch unveiled new data that only further proved that wrong by making clear that the American consumer wants their personal data protected and under their control, even if they willingly give access to AI use cases. 

The overwhelming majority of respondents were rightfully worried about how AI products will use their data. Results showed that 80% of people were concerned about AI products having access to their personal data, and 75% were worried about products being built with their personal data. Major red flags. No matter how the survey phrased the question, consumers don’t like the sound of AI leveraging personal data—and they also overwhelmingly didn’t trust Big Tech to handle it responsibly.

Another new study by Pew came to similar conclusions: The public increasingly says they don’t understand what companies are doing with their data and most believe they have little to no control over what companies or the government do with their data. 

The bottom line is that consumers are demanding companies, especially when it comes to AI, prioritize privacy with specific attention to conscientious, purpose-based data collection and retention. 

To meet this demand and privacy-proof businesses to accommodate the leaps in AI, God is in the details. Data is the precondition and the fuel for AI; AI itself begets new data. That means that tech companies must measure and mitigate the impact of data systems and their machine-learning outputs, which may have disparate impacts. If personal or sensitive information is being fed into a company’s AI, then those data inputs have to be properly permissioned for the specific use case in mind. We also need to monitor the degree to which that data persists in algorithms and their outputs. 

If dealing with sensitive personal information, the level of oversight has to be even more stringent—and it may require additional policies and monitoring systems to manage the outputs of that data. Companies also need to distinguish whether AI is being used for internal or external use cases. If internal, you need clear frameworks for sourcing training data. If external, you need to make sure you’re conducting risk assessments and mitigating downsides when they inevitably occur. 

While not exhaustive, putting some of these guardrails into place accelerates and de-risks a company’s AI initiative. Fully permissioned, privacy-safe data is the precondition for successful AI; dirty data renders otherwise well-designed AI initiatives dead on arrival. As AI products under construction hit the market, the ones that take privacy seriously will endure. The ones that take liberties with privacy will blow up on the launching pad—or if they make it off the ground, will present an exciting show of multicolored plumes streaked across the sky.


Tom Chavez is the co-founder of super{set}, a startup studio, and the chair of the Ethical Tech Project, a think-and-do tank.