Dr. Adam Rodman, an experienced doctor at Beth Israel Deaconess Medical Center in Boston, believed that chatbots powered by artificial intelligence (AI) would be a huge help for doctors in diagnosing diseases. He thought AI could give doctors a boost in their work, making them more accurate and faster. But after testing it, he was stunned by the results.
In a study Dr. Rodman helped design, doctors who had access to ChatGPT-4, a popular AI chatbot, only performed slightly better than doctors who didn’t use it. But the real shocker? The AI chatbot alone did better than the doctors.
The chatbot scored an impressive 90% when diagnosing a medical case and explaining its reasoning. In comparison, doctors using the chatbot scored around 76%, and those not using the chatbot scored 74%. This result surprised everyone, including Dr. Rodman, who was expecting the doctors to outperform the AI.
The study also uncovered some surprising insights. One of the biggest surprises was how doctors trusted their own instincts and diagnoses, even when the AI suggested something different. Many doctors struggled to fully trust the AI, even when it was suggesting a potentially better diagnosis. This highlights a key problem: while doctors are starting to use AI tools in their work, they don’t always know how to get the best results from these tools.
The main idea behind AI in medicine is that it could serve as a “doctor extender” – a tool that provides doctors with a second opinion to make sure they don’t miss anything. But this study shows that we’re still a long way from fully realizing the potential of AI in healthcare.
The Study Details
The study involved 50 doctors, including both experienced physicians and medical residents, who were randomly assigned to either use ChatGPT-4 or not. The doctors were given six real-life case histories and asked to suggest diagnoses based on the information provided. The doctors were also asked to explain their reasoning and provide possible next steps in the diagnosis.
Interestingly, the doctors who used the AI chatbot didn’t perform as well as expected. On average, they only scored slightly better than those without AI support. The chatbot, however, scored significantly higher, with an accuracy rate of 90%.
One of the test cases was about a 76-year-old man who was experiencing severe pain in his lower back, legs, and calves after undergoing heart surgery. Despite the complex nature of the case, the chatbot managed to correctly diagnose the rare condition – cholesterol embolism – which happens when cholesterol fragments block blood vessels. The doctors, on the other hand, struggled with the diagnosis and provided less accurate results.
The AI Advantage
So, what made the chatbot so good at diagnosing? The key to ChatGPT’s success lies in its ability to analyze vast amounts of information quickly and provide logical, data-driven answers. Unlike humans, AI doesn’t rely on intuition or personal experience. Instead, it makes predictions based on patterns in the data it has been trained on. And in this case, it outperformed the doctors who, despite their experience, didn’t always use the AI to its full potential.
Dr. Rodman and his team noticed that doctors were often skeptical of the chatbot’s suggestions. They didn’t always accept the AI’s reasoning, especially when it contradicted their own diagnosis. Many doctors used the AI like a search engine, asking it specific questions, rather than using it to its full potential, which would involve pasting the entire case into the chatbot and asking for a comprehensive analysis.
Why Didn’t Doctors Trust the AI?
This reluctance to trust AI is rooted in human nature. Doctors, like anyone, can be overconfident in their own expertise. Many believe that their experience and intuition are enough to make a diagnosis. But the study showed that this mindset could be holding them back, especially when AI is capable of identifying things that humans might overlook.
Additionally, many doctors still don’t fully understand how to use AI effectively. They may not realize that ChatGPT can provide detailed answers and explanations if given the right instructions. Only a small number of doctors in the study took full advantage of what the AI could offer.
The Future of AI in Medicine
The study has raised important questions about the role of AI in healthcare. While AI has shown that it can diagnose diseases better than doctors in some cases, doctors need to learn how to use AI tools to their advantage. The idea isn’t for AI to replace doctors, but rather to support them and offer a second opinion when needed.
In the future, AI could become an indispensable part of the medical field. But first, doctors need to understand how to trust and work alongside these advanced tools. For now, AI is showing promise, but there’s still work to be done before it becomes a trusted ally in diagnosing medical conditions. The question is: will doctors embrace AI as a powerful tool, or will their confidence in their own judgment continue to hold them back?
The surprising results of this study highlight a crucial shift in the medical world – AI might be the future, but the doctors’ role in embracing it will be key to making this technology truly transformative.
