Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

Need to manage AI? Start by measuring it.

Self-reporting only takes you so far.

[Photo: Getty Images]

Profile photo of Russ Fradin

CEO · Larridin

AI has moved from experimentation to broad adoption across most of the enterprise landscape. Two-thirds of organizations use AI across multiple business functions. What hasn’t moved nearly as fast is the ability to measure whether AI is working.

The same conversation keeps happening across all industry sectors: a construction company managing remote teams across hundreds of multifamily housing units. A hospital system navigating clinical and administrative complexity.

While the org charts look different, they face an identical problem—understanding where AI adoption is lacking, and where proficiency is falling short, versus where AI is driving results.

ALMOST EVERYONE USES AI, BUT FEW USE IT WELL

When executives are asked how they know their teams are using AI effectively, the answer is almost always employee surveys and subjective user self-reporting. Someone sends out a form; employees rate their own proficiency; a manager checks a box marked “advanced”; and leadership reads those numbers as signals. But they are not.

There is a meaningful difference between someone who logs into an AI tool three times a week to reformat documents versus a team that has genuinely restructured the way they work, around the technology. Self-reported surveys cannot tell you what is really happening in your organization. (Hint: If you’re not sure, it’s probably the former.)

High adoption does not mean people know how to make the best use of AI. Larridin’s proprietary data from enterprise customers puts the median AI proficiency score at 58.5 out of 100. (Based on actual, measured results, not self-reported feedback.) Organizations are adding licenses for new AI tools, running onboarding and training sessions, and aiming for full proficiency, even though the average employee is just getting started. Where is this impact most perceived? Your return on investment from AI, known as ROI AI.

BEHAVIOR AND AI MEASUREMENT

The challenge compounds for hybrid teams, which most companies have, and international teams—that is to say, for most organizations of any size. In a distributed environment, it is impossible to observe employees’ behavior as a whole, which makes companies dependent on what employees report about themselves. The issue is that what gets reported tends to be inflated. It’s not out of dishonesty, but because proficiency is genuinely hard to self-assess.

A manager in London, a team lead in Singapore, and a director in Chicago can all describe themselves as “comfortable with AI” and mean completely different things. And averaging usage scores from expert users and non-users yields a number in the middle, papering over both excellence and ignorance.

The pressure to show AI progress has also created a distortion in how organizations report upward. Boards are asking about AI readiness, and investors are factoring it into their evaluations. So no executive wants to walk into a room and say that there is widespread adoption, but no reliable way to measure what any of it is producing. So the surveys go out, the scores come back with a range of results; the average looks reasonable; and the gap between adoption, proficiency, and ROI gets wider.

3 WAYS TO TURN AI USAGE INTO REAL BUSINESS IMPACT

The real challenge is understanding whether employees can apply AI in ways that improve decision-making, productivity, and business outcomes. Leaders who want to close the growing AI proficiency gap need to start evaluating how, how much, when, and for what AI is being used in day-to-day work. Here are three practical ways to start doing that now.

1. Stop treating the rate of adoption as the primary metric.Adoption tells you how many people have access to a tool. It says nothing about whether they’re doing anything meaningful with it. Build a secondary layer that tracks the complexity and business impact of actual AI use cases, not just the number of logins.

2. Anchor proficiency estimates to observable outputs. Ask managers to submit concrete examples of AI-assisted work product, rather than rating their team’s skill level on a scale. When AI use is tied to real deliverables, self-reporting bias disappears, because the measurement shifts to what people actually produce rather than how they perceive themselves.

3. Workforce AI training only works when it’s right-sized for the people who truly need it. A staggering 85% of workers can’t connect AI training to their job. A remote operations team with low AI proficiency needs different support than a finance team running high adoption numbers, but with nothing to show for it in output quality. Diagnosing proficiency precisely separates training programs that help teams make a real impact.

    AI readiness has become a business asset, and organizations with real-time visibility into their AI landscape are better positioned to make faster, more informed decisions in every aspect of the business.

    Russ Fradin is CEO of Larridin.