AI is everywhere in healthcare now. Doctors are conflicted

[AUDIO LOGO] FLORA LICHTMAN: Hey, it’s Flora, and you’re listening to Science Friday.

SPEAKER 1: I am an endocrinologist and professor of medicine.

SPEAKER 2: I’m a psychiatrist.

SPEAKER 3: I am a hospital medicine physician.

SPEAKER 4: I am a medical student.

FLORA LICHTMAN: We’re headed to the doctor’s office, and we’re asking, how are doctors using AI in their practice? We asked our physician listeners to call in with answers, and many of you did.

SPEAKER 5: I use artificial intelligence every day in my work.

SPEAKER 6: I’ve been using it for less than half a year, and I find it to be extremely helpful.

SPEAKER 7: One of the ways I use it is when a patient asks me a question directly in their electronic chart, and it automatically generates a reply that I could potentially send back to the patient. Sometimes the AI version of the answer is much more polite than my version would be, oftentimes.

FLORA LICHTMAN: What we heard from doctors mirrored broad trends, that more and more doctors are using AI tools for diagnosis, paperwork, and getting up to speed on patients. So how is AI changing medical care? And how is it affecting patients? Here to sort fact from hallucination is Dr. Jonathan Chen, Director for Medical Education in Artificial Intelligence at Stanford University. He studied the accuracy of these models and how health workers use them. Jonathan, welcome to the show.

JONATHAN CHEN: Thanks so much for having me, Flora. Looking forward to a very dynamic conversation that’s changing under our feet.

FLORA LICHTMAN: Me, too. Can you give me a sense of how dynamic this is, how much the daily workflow has changed for doctors in the past year or two?

JONATHAN CHEN: I liken it to it’s like the internet got invented three years ago. We’re rapidly trying to adapt to it. I worked in this space, AI and medicine, 10 years ago, before it was cool. And six years ago, I had students working on this. This stuff was unusable. You wouldn’t even bother talking about it.

Three years ago, ChatGPT blows up on the scene. It’s like, whoa, this can barely pass a medical exam. But it’s barely passing. That’s things not smarter than me. And now it’s like, no, it’s basically as good or better than most doctors would be answering medical questions. And we’re still figuring out how to integrate that in a responsible and effective way, while managing very predictable harms at the same time.

FLORA LICHTMAN: How many doctors in the US are using AI for their work? Do we know?

JONATHAN CHEN: I bet a lot of them are using it without even realizing it. If you just go into Google search, it is starting to integrate AI into those responses, whether even intended to or not. So more and more, I think the majority are starting to do it. Two years ago, when we did a study, maybe a third had never touched it before, a third used it once or twice, and maybe a third of docs had used it years ago. But now it’s clearly the majority are starting to figure out, at least have touched it. But that doesn’t mean they know how to use it effectively. That’s still an ongoing battle to figure out.

FLORA LICHTMAN: What are the tools or platforms that are most often used by doctors?

JONATHAN CHEN: The reality is just the generic chat platforms, your ChatGPT, your Claudes, your Geminis. Within medical specific ones, several have started to blown up. How do you combine this great chat capability, but with actual medical specific knowledge references? So now we just had a study recently. AMBOSS came out. There’s this German company. Glass Health’s on there, but also Doximity has a product.

And OpenEvidence is certainly the company that blew up out of nowhere in the past several years and became a multi-billion dollar company trying to create this interface in the form that doctors are used to. And it is pervasive all over the place when, three years ago, you’ve never even heard of these things before.

FLORA LICHTMAN: What are doctors using this chat feature for? What kinds of questions are they asking? And are there other uses?

JONATHAN CHEN: For one, there are also these ambient scribe tools like, let’s just listen to our conversations. Your doctor’s not spending their time writing into their note. They could just be talking to you or looking into your face, so there’s that kind of thing. A lot of these ones, I think, are more compelling because it’s really answering questions. They’re really almost like a second consultation.

We used to, as doctors, look up an article in a medical encyclopedia and try to figure out if it applies to you. I see a lot of doctors and trainees I work with, they don’t want to look up an article and then read it and figure out if applies to you. You just want an answer now, which means you ask an AI system, it’ll read the article for you and say, I read the article. This looks like the answer. And that’s a different dynamic and a different interface and I think why people are really gravitating toward and really getting hooked on it. I don’t really see how we’re going back.

FLORA LICHTMAN: What are the problems that these tools are solving for doctors?

JONATHAN CHEN: Oh, gosh. Solve is actually is a higher bar. It’s certainly assisting with a lot of things. But making sense of and organizing medical information, keeping track of things is actually very powerful. A very common human behavior, doctor behavior– it’s like, we’re all very smart, but there’s unlimited medical knowledge. So we’ll often run into our colleague. Hey, what’s the latest cholesterol guidelines?

I literally had a patient ask me, if a patient has a hip fracture, and there’s some blood around the joint, does that need surgery? I’m not a surgeon. I don’t really know. But could I run into– a curbside console is a classic phrase, where I run into you on the curbside. Can I ask you a quick question? That’s the kind of thing.

The reality is, AI is pretty freaking good at this kind of thing. And is it perfect? No, but it’s also way more accessible in ways that just wouldn’t be practical for doctors or patients.

FLORA LICHTMAN: So if a doctor was looking for help with a curbside consultation, and they opened OpenEvidence, how is that query or that process different from my experience in ChatGPT?

JONATHAN CHEN: The reality is it’s grounded in the same underlying technology, the large language model technology that’s reading your phrase and digesting the internet, autocomplete on steroids, guesses the next word. What a lot of these medical ones are focusing on is– I mean, if a chatbot says, how do you prescribe morphine, who knows where it got that from the internet. You don’t really want to trust that. But here, they combine it with RAG, Retrieval Augmented Generation.

What does that mean? They’ll find an article, a medical guideline. Ah, this seems to be talking about your question. And it will read sections of that article and tell you what the answer is, which is nice, because then you can trust, but verify. We feel a lot better if we can point back to a source, and we can say, ah, I can go back and look at this directly.

FLORA LICHTMAN: What about hallucination? Is that a problem you have to worry about? Or making up citations?

JONATHAN CHEN: It is a very real issue. I won’t say that’s a solved problem, but it’s a lot better than it was two or three years ago. Two or three years ago, it was a huge issue. These chatbots would confidently make up citations, and that’s worse than if it’s wrong, because it’s so disarming. It so has the appearance of credibility. And hallucinations and confabulations can very easily lead you the wrong way.

And there have been studies that show people are sometimes worse off using AI because they are confidently following a wrong path or something that looks so believable. I won’t say it’s a solved problem, but these RAG-based systems, the ones that look up articles for you– and really, you can click on the link and go to the source. It helps a lot.

I would say now you have different kinds of issues. What if the question doesn’t have an answer in the literature? Many different kinds of studies are higher quality than others. What if it cites something and doesn’t really interpret it in the right way or interprets a low quality study? These are very real tensions that it’s not an AI problem. It’s just a fundamental medical knowledge and reasoning problem, that this is a danger because it looks so good.

Oh my gosh, it looks so good. People treat these things like they’re the oracle, like they’re God. And I had to tell my trainees, this thing is not God. It is very powerful. It’s really cool. But just because it says something does not mean you can blindly trust it.

FLORA LICHTMAN: I want to get back to trainees in a second, but we asked our listeners as patients, how do they feel about their doctors using AI, and we got a variety of responses.

SPEAKER 8: They ask before they use it, is it OK? And I think that’s pretty important.

SPEAKER 9: I feel a lot of times big skepticism when I see things– AI will fix this. But maybe this is a good thing. Let’s wait and see.

SPEAKER 10: I don’t think it’s ready for the medical system.

FLORA LICHTMAN: OK. So here’s my question, Jonathan. Have we done the research to know if these tools, which you say the majority of doctors are using, are helping patients, are doing more good than harm?

JONATHAN CHEN: Ooh, that’s a deep question. There are some very interesting studies– not mine, others who study regular people using ChatGPT or other things to answer medical questions. And the AI made patients actually worse. In our studies, it sometimes helps doctors, sometimes doesn’t help them enough.

But in patient cases, sometimes it made them worse because it’s not just having the knowledge. With a great podcast interview, it’s how do you elicit the information, frame the question, and interpret the answer the right way? If you don’t do that well, it’ll go way off the rails.

FLORA LICHTMAN: Yeah. Well, I wanted to ask you about this. We know prompts matter so much in what you get out of AI. Are doctors getting trained to use these tools?

JONATHAN CHEN: At the high level, no. When we submitted one of our studies a couple of years ago showing what happens when doctors use ChatGPT in this case, a peer review said, that’s not realistic. No one would unleash some computer system for doctors to use without properly training them first. I’m like, what are you talking about? That is exactly what they do. That happens all the time. It’s happening right now.

So most clinicians, patients are not getting any formal training in how to use these things. One of my jobs now is the newly created role, Director for Medical Education in AI at Stanford, but specifically because we need more of that here and broadly. And it is letting them, the trainees and the doctors, know these are real tools. You need to understand a little about how they work, because they’re clearly going to become a part of your life and your work. Know what the caveats are so you can use them safely, effectively, and responsibly.

FLORA LICHTMAN: What about procedures? We know hospitals and medical systems have tons of procedures, often to cover their butts in terms of liability. Are hospitals or medical centers actively writing new AI usage policies?

JONATHAN CHEN: They are, and it’s a tough thing. Let alone scientific peer review usage policies, they’re lagging. The technology is moving so fast. It’s crazy. It’s moving. Every month, it’s like, shoot, another update. And your policy you wrote six months ago is already out of date.

A couple years ago, a lot of people are trying to ban the technology. We don’t how to deal with it. It’s scary, so just ban it. Nobody’s allowed to use it. That’s completely impractical, even if it were a good idea, which I don’t think it is. That’s my opinion, and there’s not consensus.

You cannot effectively ban it because someone can just pull out their phone, and work around you. And now they’re probably using a non-secure, non-privacy-compliant tool to do what they would have done anyway. So more of these policies, more of these frameworks are coming out. But it’s happening so haphazard, so distributed, because the technology has just moved so fast that human institutions cannot keep up pace with how fast it’s moving.

FLORA LICHTMAN: After the break, I want to talk to you about medical students and trainees and how they should be using these tools. You down to stay with us?

JONATHAN CHEN: Absolutely.

FLORA LICHTMAN: Don’t go away. One thing that was interesting in the calls we got from doctors is that even though almost everybody who called us said they were using it, they also were conflicted about it for interesting reasons.

SPEAKER 11: It is definitely something that we are learning about in medical school, because it’s something that we’ll be using in our futures as physicians.

SPEAKER 12: I do worry that learners will become too reliant on generative AI, and as a result, they may lose some of their clinical reasoning skills. And I also worry about biases that are inherent in the models that AI is trained on.

FLORA LICHTMAN: Are you worried about this, that med students are going to be overly reliant? They won’t be able to vet the information they’re getting from these tools?

JONATHAN CHEN: It is a real issue, a real issue. I say there’s a very huge dilemma, and we have absolutely not consensus on this. We recently did a poll, and 60% in favor, 40% against. We had a debate, and then it went to 55% in favor. And you still need a policy on what to do.

I’ll give this brief anecdote to really punctuate the point. In our medical reasoning class at Stanford– here’s a case. What do you think the diagnosis is? What do you think the treatment and management should be? On the homework two years ago, the students are killing it. They’re killing it. But when it came time for the closed book exams, twice as many students failed that exam compared to usual. What happened?

It’s obvious what happened. They’re using the AI to do their homework, and then they miss the point. You’re supposed to struggle through homework so that you then actually learn and are ready for the closed book exam or the live patient interaction, where you can’t have AI do it for you. So that’s definitely a real danger.

On the other hand, when we cautioned them last year, this is a very powerful tool– the best tutor you’ve ever had if you use it. But like a chainsaw, it’s a powerful tool that you could really hurt yourself with if you don’t use it properly. This time, the students actually did better on the closed book exam, and they learned to use it in practice when they are understanding how to use it with the right guardrails.

FLORA LICHTMAN: We’ve been talking a lot about chatbots and querying for medical questions. But what about AI for analyzing radiology? Where are we with that use of AI in medicine?

JONATHAN CHEN: Sure. Computer vision, actually, I would say, is much more mature. That was a very hot thing five, six, seven years agao. It was here, look through this X-ray. Tell me if I have a lung infection. There’s a nice study where doctors using a thing to help it find polyps on a colonoscopy, it made them better. And when they turned off the AI computer vision tool, the doctors were worse than before they started. They got used to it. They started to depend on that technology, so very powerful capabilities.

But the famous thing here is Geoffrey Hinton, Nobel Prize winner, a father of a lot of neural networks. He famously said about seven or eight years ago, it’s so obvious all radiologists should just stop working. They’re all going to be replaced by computers within five years. He said this eight-ish years ago. Clearly he’s wrong, because now there’s more radiologists needed than ever before. But it was a shifting and understanding of what the jobs and needs are. Spotting things in an X-ray, maybe that’s not the task anymore. It’s synthesizing, organizing that, and managing that for increasing demand for these services, too.

FLORA LICHTMAN: Hm. Do you worry about privacy concerns? Are these tools HIPAA compliant? Do we know how the patient data is getting used?

JONATHAN CHEN: So it’s a huge issue, and that’s actually one of the key trainings we give our people, doctors and trainees– FYI, you cannot put in real patient information into ChatGPT or Claude or Gemini or whatever. You have totally just uploaded private patient information to a public server when you do that. And some tech company now knows all of that. It’s very easy for that to be invisible and people to not notice that. So that’s been a lot of the guardrails in place to manage this.

But the companies also have to be responsible for where it can go wrong. Right now, there’s too thin a really fake disclaimer. FYI, you should not use this for actual medical advice, even though I know you’re actually doing that. And literally, companies will put out press releases. Look at this cancer patient. We’re saving their life. They’re really talking out of both sides of their mouth. It’s like, OK, if you want to provide the benefit, you’ve got to take the responsibility. But if something goes bad, literally somebody can sue me for medical malpractice.

FLORA LICHTMAN: Are you liable or is the tech company liable? If you use one of these tools, like OpenEvidence or something else, you rely on it for medical advice. You pass it along, something goes really wrong. Who gets sued?

JONATHAN CHEN: Yeah, it’s a very thorny issue. Well, if you’re malpractice, what you do is you sue everybody and you see where it sticks. Right now, in theory, it sticks to the doctor. All these tools, at the end of the day, I signed the order. And so in theory, I’m the one who gets sued. I’m like, dude, I use 20 different tools. I don’t even how all of them work. This magical reference library, it seemed to have reasonable information.

So that actually is an awkward thing, that they have very limited responsibility. The FDA, Health and Human Services, many entities are trying to figure out how to wrap their heads and their brains around this, realizing it’s a very difficult problem and we have to figure out how to adapt to it.

FLORA LICHTMAN: Well, speaking of a magical reference library, I read that you’re also a magician. Do you see any relationship between this work and that work?

JONATHAN CHEN: It is. Somewhat, it was a coincidental hobby that span out of control. But it’s really aligned. And being a good magician is actually about having good empathy. You have to understand what another person is thinking so that you can trick them with the other thing and get them to think another way. And so much of AI is like, it’s so believable, but it can’t be real. This thing looks like it expresses emotion.

The caller said, it’s more polite. It’s not polite, it’s not kind. It doesn’t think. It’s a computer. It doesn’t do anything. But man, is that illusion really convincing. And if you want to operate in the world effectively, you have to be able to distinguish. You still need your judgment, not your knowledge. You need your judgment to distinguish what is real and what is not so that you can do some good without hurting yourself or others.

FLORA LICHTMAN: Dr. Jonathan Chen is an associate professor of medicine at Stanford University. Thank you for taking the time to talk to us today.

JONATHAN CHEN: Great to talk to you. I look forward to catching up.

FLORA LICHTMAN: This episode was produced by Dr. Peter Schmidt. Speaking of questions, we’re working on a segment about perimenopause, and if, like me, this topic is dominating your group chat, I want to hear a story about how you feel like perimenopause is affecting you. Brain fog, rage cleaning, a desire to run away from your family and make a new life in a motel room– whatever it is, I want to hear the story. We’ll have an expert to sort out the science behind the symptoms we hear so much about, at least on social media. 877-4-SCIFRI is our number.

Thanks for listening. We’ll catch you next time. I’m Flora Lichtman.

Copyright © 2026 Science Friday Initiative. All rights reserved. Science Friday transcripts are produced on a tight deadline by 3Play Media. Fidelity to the original aired/published audio or video file might vary, and text might be updated or amended in the future. For the authoritative record of Science Friday’s programming, please visit the original aired/published recording. For terms of use and more information, visit our policies pages at http://www.sciencefriday.com/about/policies/