Who performed better when giving medical advice? The physician or the AI chat bot?

https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309?guestAccessKey=6d6e7fbf-54c1-49fc-8f5e-ae7ad3e02231&utm_source=For_The_Media&utm_medium=referral&utm_campaign=ftm_links&utm_content=tfl&utm_term=042823

This is hot off the press on April 28th, 2023. Reddit has a sub forum where people can ask medical questions. Physicians and an AI Chat Bot both answer questions that patients ask. So who (or what) performed better?

“ In this cross-sectional study of 195 randomly drawn patient questions from a social media forum, a team of licensed health care professionals compared physician’s and chatbot’s responses to patient’s questions asked publicly on a public social media forum. The chatbot responses were preferred over physician responses and rated significantly higher for both quality and empathy.” Well look at that. A ROBOTIC response was more empathetic than a LIVE HUMAN physician.

“A public and nonidentifiable database of questions from a public social media forum (Reddit’s r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose “which response was better” and judged both “the quality of information provided” (very poor, poor, acceptable, good, or very good) and “the empathy or bedside manner provided” (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians.”

Of the 195 questions and responses, evaluators preferred CHATBOT responses to physician responses in 78.6% of the 585 evaluations. Physician responses were significantly shorter than chatbot responses 52 [17-62] words vs 211 [168-245] words. Chatbot responses were rated of significantly higher quality than physician responses. The proportion of responses rated as good or very good quality was higher for chatbot than physicians (chatbot: 78.5%, physicians: 22.1%). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses. The proportion of responses rated empathetic or very empathetic was higher for chatbot than for physicians (empathetic: physicians: 4.6%, chatbot: 45.1%. This amounted to 9.8 times higher prevalence of empatheticor very empathetic responses for the chatbot.

This data table presents an interesting point as well. CHATBOT has no very poor quality ratings, and minimal poor ratings. They peak much higher when physicians start to drop off in the good rating. In terms of empathy, it is nearly a mirror image with moderate empathy being the center.

A few points to make here. One, it would appear that it is only physicians in the subreddit AskDocs forum. I am guessing there are no PA’s or APRN’s in that forum answering questions. So this is purely physician versus chatbot. Second, if we wondered how disgusting healthcare has gotten, this is an interesting way of pointing it out. If a robot can be programmed to give a higher quality answer and be more compassionate, what does that say about the standard of care provided by physicians these days? It would be interesting to compare different models of healthcare answers versus chatbot. I would love to see a comparison for APRN’s and PA’s as well. I would speculate the APRN’s would come in much more competitively in the empathy and the quality of detail to the answer. Those two areas are where the difference in training versus MD’s really shines through.

I wonder if the Reddit physicians will change their approach after reading this?