Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

Open evaluation is the only way to trust medical AI

Building it is the hard part.

[Photo: Getty Images]

Profile photo of David Talby

CEO · John Snow Labs

Passing a medical licensing exam means little when predicting whether an artificial intelligence model can safely perform real-world clinical work. Yet vendor pitch decks routinely cite standardized test scores as proof of clinical readiness.

For healthcare to adopt AI safely at scale, evaluation must be open, reproducible, and dynamic. But building trustworthy healthcare benchmarks is uniquely difficult. The industry faces structural data barriers, benchmark contamination, and the growing complexity of autonomous AI agents. Addressing these challenges is essential to responsible clinical AI adoption.

THE CASE FOR OPEN, EXPLAINABLE SCIENCE

A benchmark score is only trustworthy if its methodology is transparent and independently reproducible. Consider MedHELM (Medical Holistic Evaluation of Language Models), a clinician-validated benchmark spanning 121 clinical tasks across five categories using a public methodology.

Yet MedHELM illustrates healthcare evaluation’s fundamental obstacle: The most valuable data often cannot be shared. Of its 35 underlying datasets, 14 are private, and another seven are access-restricted. Independent researchers cannot fully reproduce published model rankings without obtaining access to protected patient records.

This problem extends across healthcare AI. MIMIC-IV, a widely used clinical dataset, requires individual credentialing, ethics training, and a restrictive data use agreement. It also represents a narrow clinical population: records from one Boston academic medical center, primarily covering emergency and intensive care between 2008 and 2022. Outpatient settings, community hospitals, and broader regional populations are underrepresented.

These restrictions protect patient confidentiality and support compliance with HIPAA, the Health Insurance Portability and Accountability Act. But they also prevent healthcare benchmarks from becoming easily distributable, open-source evaluation suites that enterprises can independently run before deployment.

One promising workaround is high-fidelity synthetic data, validated against real clinical records for task difficulty, scoring distributions, and model rankings. Creating and maintaining such datasets, however, requires sustained investment.

THE CONTAMINATION CLOCK IS TICKING

Even accessible benchmarks face another threat: data contamination.

When public evaluation questions enter foundation-model training datasets, benchmark scores can become misleading. Models may perform well partly because they have encountered the questions before, rather than because they can generalize to unfamiliar clinical scenarios.

Research on benchmark contamination reinforces this concern. “An Open-Source Data Contamination Report for Large Language Models” in Findings of the Association for Computational Linguistics documented contamination across widely used benchmarks, with measurable performance advantages associated with contaminated evaluation data.

This creates a difficult trade-off. Keeping test sets private helps protect their integrity but limits independent verification. Publishing them enables transparency but increases exposure to future training datasets.

The solution is not simply choosing openness or secrecy. Healthcare needs continuously refreshed evaluations, protected holdout datasets, and transparent testing methodologies that can detect when benchmarks become compromised.

EVALUATE ACTIONS, NOT ANSWERS

As healthcare AI evolves from question-answering systems into autonomous agents, evaluation becomes substantially more complex.

An agent reviewing patient charts, navigating electronic health records, placing orders, or submitting prior authorizations must do more than produce accurate answers. Evaluators must determine whether it formulated the right clinical plan, selected appropriate tools, handled unexpected errors, and completed the entire workflow safely.

Emerging benchmarks demonstrate how much work remains.

  • PhysicianBench: Evaluates AI agents performing 100 complex physician tasks adapted from specialist consultations across 21 medical specialties in simulated EHR environments. In the original evaluation, the best-performing model successfully completed just 46% of tasks, exposing the gap between medical knowledge and reliable clinical execution.
  • HealthAdminBench: Tests 135 healthcare administrative tasks involving prior authorizations, claims appeals, and other workflows across simulated EHR, payer portal, and fax interfaces. The best-performing agent achieved only 36.3% end-to-end task success, underscoring the difficulty of coordinating complex operations across fragmented systems.

Interactive evaluations offer an additional advantage: Dynamic, multi-step tasks are harder to memorize than static questions, potentially reducing contamination risks.

But building realistic simulations, maintaining clinical accuracy, and evaluating complete action sequences demands far greater infrastructure and investment than conventional question-answer benchmarks.

BEYOND THE PITCH DECK

Healthcare organizations cannot treat AI evaluation as a one-time procurement or governance checkbox. Leaders should ask vendors four essential questions:

1. Task relevance: Do your benchmark scenarios reflect our actual clinical and administrative workflows?

2. Freshness and contamination: When was the benchmark last updated, which model versions were tested, and how do you prevent data leakage?

3. Reproducibility: Can our organization or an independent third party reproduce your reported results?

4. Behavioral testing: Does your evaluation measure static knowledge, or can it verify safe, end-to-end execution in realistic environments?

No single company or academic institution can overcome healthcare’s privacy constraints, contamination risks, and agentic evaluation challenges alone.

Trustworthy AI evaluation requires shared investment from healthcare systems, researchers, and technology vendors. By collectively building and maintaining open, reproducible, and dynamic benchmarks, the industry can move beyond impressive test scores toward demonstrably safe, effective clinical AI.

David Talby is CEO of John Snow Labs.