Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

What marketers get wrong about A/B testing

Most are running an expensive coin flip.

[Photo: Getty Images]

Profile photo of Pritam Roy

Co-Founder and Chief Product Officer · Fibr AI

There’s a number that should bother every marketing leader more than it does: 70% to 80%. That’s the share of A/B tests that are either inconclusive or are stopped before reaching statistical validity. Most results are declared significant before the data has said anything, according to an analysis of more than 28,000 marketing experiments.

Marketers build entire programs on A/B testing their websites and call it optimization.     

But A/B testing still serves the same page to everyone in a given variant, it’s a tool for choosing between static experiences, not transcending them. The question I keep returning to is why do we treat websites as static      when everything around them responds to user behavior in real time? Your paid media adjusts in real time. Your CRM learns. Your email targeting evolves. But the website, where every dollar of acquisition spend ultimately lands, still serves every visitor the same undifferentiated page. At scale, that’s not just inefficiency. It’s structural waste.

The illusion of the winning variant

Here’s a scenario that plays out inside most large marketing organizations: a test runs, a variant wins, the team ships it, and everyone moves on. The next experiment starts from scratch. No memory of what worked. No structured connection to what comes next.

We call this experimentation. A more accurate term is episodic event generation.

When a variant wins, most programs record an outcome: B beat A. But they fail to encode the actual intelligence embedded in that result—which cohort responded, which traffic source drove the visit, and under what conditions the result held. Without that structure, you’re not building institutional knowledge. You’re running an expensive coin flip.

If your program’s only output is “Variant B won,” you’re extracting a fraction of what’s contained in that data. A meta-analysis of 115 real-world A/B tests puts the median lift at around 4% to 5% for tests that actually reach significance.

The hidden cost of traditional A/B testing

Running a single experiment properly—engineering time, QA, legal review in regulated categories, analyst interpretation—costs organizations upwards of $20,000 per test. That’s before the opportunity cost of traffic sent to an underperforming variant while waiting to collect enough data to be statistically significant. It’s why most enterprise teams, even well-resourced ones, are running fewer than 100 experiments a year. The cost structure makes scale impossible.

The bottleneck isn’t ideas. It’s the cost and time required to convert an idea into a live test. Most of what gets discovered lives in spreadsheets and quarterly readouts. It doesn’t persist as system logic. It walks out the door when someone changes teams.

From one experiment to 100

Think about what’s happening in parallel on any meaningful website: thousands of traffic segments, dozens of acquisition channels, multiple device contexts, seasonal behavioral shifts. Running experiments serially is like tuning an aircraft engine one bolt at a time while it’s in flight.

Start with one URL. Apply intelligent audience segmentation across four or five distinct cohorts. Generate 10 hypotheses per audience. Run 10 variants per experiment. You haven’t added headcount but you’ve generated 100 times the experimental throughput of a traditional program, feeding a single learning system.

When you build it that way, hypothesis 50 is more likely to produce signal than the first hypothesis, not because the team got better, but because the system accumulated context.

Stop sending visitors to an underperforming variant

Traditional A/B testing splits traffic uniformly: 50 percent to control, 50 percent to variant. That’s a sensible design when the goal is measurement. It’s a poor design when the goal is conversion. For the entire duration of a test, you’re knowingly sending a large share of visitors to an experience you already suspect may underperform.

Dynamic allocation approaches, such as multi-armed bandit optimization, fix this by continuously routing traffic toward variants that are performing and away from those that aren’t. Faster convergence, fewer users exposed to underperforming variants, and more conversion value from the same traffic.

Every experiment should inform the next

A winning variant is a signal about a segment and a moment in the user journey. That context needs to be tagged and stored alongside the test result, and when something works for a given audience, it should automatically propagate to similar pages across the site, not wait in a slide deck for someone to remember to act on it.

Every experiment should inform the design of the next one. And without continuous monitoring against a live baseline, you’re making permanent decisions from temporary data.

When experimentation is architected as a learning system, the math changes entirely. A 4% lift from a quarterly experiment is meaningful. A system that accumulates those gains iteratively, across segments, across traffic sources, continuously, produces 4% on top of 4% on top of 4%. That’s the difference between a program and a machine.

The questions that matter

Two recent signals point in the same direction. Google has patented the ability to replace your landing page when its AI determines it can convert better than yours. When that happens, your tag manager doesn’t fire. Your customer data platform doesn’t capture the session. Google’s platform controls the experience and keeps the intelligence.

Meanwhile, OpenAI’s $1.1 billion acquisition of Statsig signals that experimentation is becoming core infrastructure for intelligent systems, not just a marketing function. Every paid click landing on a static page funds the dataset being used to intermediate your customer relationship further.

This raises a different set of questions: Does your website get smarter with every visit, or just every quarter? If your experimentation lead left tomorrow, would the system retain its intelligence? Are you learning fast enough to compete with systems that improve continuously? For most large programs, the honest answer is damning. Throughput is a fraction of what the traffic volume warrants. Allocation logic is still manual, static, and quarterly.

The experiments you’re running today are trying to optimize a static page. The real opportunity is making the page itself into a system that learns. One that runs at scale, allocates dynamically, and compounds continuously.

That’s not a tools problem. It’s an architecture problem. And the organizations that recognize the difference are the ones whose conversion curves will show it before platforms make decision for them.

Pritam Roy is co-founder and chief product officer of Fibr AI.