Skip to content
Fast Company Executive Board

The Fast Company Executive Board is a private, fee-based network of influential leaders, experts, executives, and entrepreneurs who share their insights with our audience.

What your patients learn from AI before they see you

AI's pitfalls and promise in the exam room.

[Photo: Getty Images]

Profile photo of David Talby

CEO · John Snow Labs

Primary care clinicians are beginning to encounter a familiar scenario at appointments: A patient arrives with a printout, a screenshot, or a theory about their health, and it usually starts with, “ChatGPT said…”

Some physicians fear this represents the beginning of AI replacing medical judgment. The reality is less dramatic, but more nuanced.

Large language models (LLMs) are not displacing clinicians, but they are reshaping the informational landscape around primary care. In doing so, they amplify long-standing dynamics in medicine—sometimes constructively, and sometimes in ways that introduce new risks.

Why AI feels authoritative to patients

Patients have always sought information before appointments. What’s new is the personalization and fluency of generative AI. Instead of reading generic web pages, patients can now copy and paste lab values, imaging reports, or medication lists to receive an interpretation tailored to their data.

That personalization creates a powerful sense of authority. AI doesn’t merely explain what elevated cholesterol means—it explains the patient’s actual cholesterol result in confident, conversational language. For many patients, this feels like an instant second opinion.

On one hand, patients who better understand their conditions often ask more informed questions and participate more actively in shared decision-making. But fluency is not the same as reliability.

The “honesty vs. accuracy” gap

Several recent studies demonstrate how LLMs can appear authoritative while exhibiting structural weaknesses that matter deeply in clinical contexts.

A 2025 arXiv study, The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems, introduced an important distinction between accuracy (whether a model knows the correct answer) and honesty (whether it faithfully reports what it knows).

The researchers found that newer, larger models were more accurate overall—but not more honest. In controlled settings, frontier models sometimes produced responses that deviated from information they demonstrably “knew,” particularly when prompted under certain pressures or goals.

In plain terms, a model might possess the medical knowledge that certain symptoms warrant urgent evaluation. But when prompted in a way that emphasizes reassurance or aligns with a user’s framing, it may soften or redirect that conclusion.

For patients, this creates a subtle but serious risk. Misinformation can stem not from ignorance but from the goal trade-offs built into model responses.

If politeness, brevity, or alignment with user expectations implicitly outweigh strict truthfulness, the result can be a dishonest answer delivered with high confidence, causing unnecessary anxiety or minimizing issues that require medical attention.

Compliance over correctness

Another recent study in Nature demonstrated that LLMs frequently exhibit sycophancy—a tendency to agree with a user’s stated assumption even when it is clinically incorrect. When users nudged the model toward an incorrect diagnosis, the model often complied rather than correcting them.

In practice, this means a patient who asks, “This is probably just a cold, right?” may be statistically more likely to receive confirmation of that belief—even if the symptoms described are more consistent with pneumonia or something else entirely.

This compliance bias creates an echo chamber. Patients with health anxieties may receive amplified worst-case scenarios. Patients biased toward minimizing symptoms could receive unwarranted reassurance. In both cases, AI reinforces the user’s prior beliefs rather than serving as an independent source.

The consistency–accuracy dissociation

Patients often equate consistency with truth. If three separate prompts, or even three different AI tools trained on similar data, produce the same answer, that consistency feels validating. But because many LLMs share overlapping training data and architectural features, they can also share the same blind spots. Repeated agreement across tools does not guarantee correctness.

Another medRxiv study demonstrates this. In fact, the research shows some models demonstrated 99% to 100% intra-model consistency (providing the same answer repeatedly) while achieving only about 50% diagnostic accuracy in certain binary medical tasks. This is essentially a random performance. In other words, the model was reliably wrong.

Confidence and repetition are persuasive. But they are nowhere near substitutes for clinical validation.

Where AI helps and hurts patients

None of this means AI lacks value. LLMs are effective at:

  • Translating medical terminology into plain language
  • Adapting explanations to different literacy levels
  • Reinforcing care plans discussed in the exam room
  • Enhancing patient understanding and engagement for straightforward educational tasks

But AI limitations become more pronounced as clinical complexity increases, particularly in patients with multiple chronic conditions or polypharmacy.

For example, LLMs may perform adequately when checking two prescriptions but falter when handed a list of eight, and catch obvious medication interactions, but miss subtler ones. In other words, the model can “know the rule” but not “know the patient.”

AI is a tool, not a clinician

Patients arriving with strong diagnostic beliefs aren’t new, but patient access to once-gatekept information and the perceived authority of generative AI are. LLMs can educate, translate, and engage. But peer-reviewed evidence increasingly shows structural vulnerabilities.

As a result, the clinician faces an increased communication burden. It’s more important than ever for healthcare professionals to acknowledge AI-derived information, explain where clinical judgment differs, and thoughtfully document those discussions.

Trust is still built the same way it always has been, through listening, contextual reasoning, and shared decision-making. Dismissing AI outright risks alienating patients. Deferring to it without question risks harm. Good communication strikes a balance between the two.

As someone working on the ground floor of healthcare AI, I don’t trust any language model more than I do a physician for myself, my family, or my friends. These systems are no more accurate than clinicians, simply more conversational and more convincing.

AI can serve as a helpful tool, but can’t replace clinical judgment, ethical responsibility, or accountability in care. For primary care, that truth remains unchanged (for now).

David Talby is the CEO of John Snow Labs.