stockyardsremolding

Last year, headlines describing a study about artificial intelligence (AI) were eye-catching, to say the least:

• ChatGPT Rated as Better Than Real Doctors for Empathy, Advice
• The AI will see you now: ChatGPT provides higher quality answers and is more empathetic than a real doctor, study finds
• Is AI Better Than A Doctor? ChatGPT Outperforms Physicians In Compassion And Quality Of Advice

At first glance, the idea that a chatbot using AI might be able to generate good answers to patient questions isn’t surprising. After all, ChatGPT boasts that it passed a final exam for a Wharton MBA, wrote a book in a few hours, and composed original music.

But showing more empathy than your doctor? Ouch. Before assigning final honors on quality and empathy to either side, let’s take a second look.

What tasks is AI taking on in health care?

Already, a rapidly growing list of medical applications of AI includes drafting doctor’s notes, suggesting diagnoses, helping to read x-rays and MRI scans, and monitoring real-time health data such as heart rate or oxygen level.

But the idea that AI-generated answers might be more empathetic than actual physicians struck me as amazing — and sad. How could even the most advanced machine outperform a physician in demonstrating this important and particularly human virtue?

Can AI deliver good answers to patient questions?

It’s an intriguing question.

Imagine you’ve called your doctor’s office with a question about one of your medications. Later in the day, a clinician on your health team calls you back to discuss it.

Now, imagine a different scenario: you ask your question by email or text, and within minutes receive an answer generated by a computer using AI. How would the medical answers in these two situations compare in terms of quality? And how might they compare in terms of empathy?

To answer these questions, researchers collected 195 questions and answers from anonymous users of an online social media site that were posed to doctors who volunteer to answer. The questions were later submitted to ChatGPT and the chatbot’s answers were collected.

A panel of three physicians or nurses then rated both sets of answers for quality and empathy. Panelists were asked “which answer was better?” on a five-point scale. The rating options for quality were: very poor, poor, acceptable, good, or very good. The rating options for empathy were: not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic.

What did the study find?

The results weren’t even close. For nearly 80% of answers, ChatGPT was considered better than the physicians.

• Good or very good quality answers: ChatGPT received these ratings for 78% of responses, while physicians only did so on 22% of responses.
• Empathetic or very empathetic answers: ChatGPT scored 45% and physicians 4.6%.

Notably, the length of the answers was much shorter for physicians (average of 52 words) than for ChatGPT (average of 211 words).

Like I said, not even close. So, were all those breathless headlines appropriate after all?

Not so fast: Important limitations of this AI research

The study wasn’t designed to answer two key questions:

• Do AI responses offer accurate medical information and improve patient health while avoiding confusion or harm?
• Will patients accept the idea that questions they pose to their doctor might be answered by a bot?

And it had some serious limitations:

• Evaluating and comparing answers: The evaluators applied untested, subjective criteria for quality and empathy. Importantly, they did not assess actual accuracy of the answers. Nor were answers assessed for fabrication, a problem that has been noted with ChatGPT.
• The difference in length of answers: More detailed answers might seem to reflect patience or concern. So, higher ratings for empathy might be related more to the number of words than true empathy.
• Incomplete blinding: To minimize bias, the evaluators weren’t supposed to know whether an answer came from a physician or ChatGPT. This is a common research technique called “blinding.” But AI-generated communication does not always sound exactly like a human, and the AI answers were significantly longer. So, it’s likely that for at least some answers, the evaluators were not blinded.

The bottom line

Could physicians learn something about expressions of empathy from AI-generated answers? Possibly. Might AI work well as a collaborative tool, generating responses that a physician reviews and revises? Actually, some medical systems already use AI in this way.

But it seems premature to rely on AI answers to patient questions without solid proof of their accuracy and actual supervision by healthcare professionals. This study wasn’t designed to provide either.

And by the way, ChatGPT agrees: I asked it if it could answer medical questions better than a doctor. Its answer was no.

We’ll need more research to know when it’s time to set the AI genie free to answer patients’ questions. We may not be there yet — but we’re getting closer.

Want more information about the research? Read responses composed by doctors and a chatbot, such as answers to a concern about consequences after swallowing a toothpick.

About the Author

Robert H. Shmerling, MD, Senior Faculty Editor, Harvard Health Publishing; Editorial Advisory Board Member, Harvard Health Publishing

Dr. Robert H. Shmerling is the former clinical chief of the division of rheumatology at Beth Israel Deaconess Medical Center (BIDMC), and is a current member of the corresponding faculty in medicine at Harvard Medical School. … See Full Bio View all posts by Robert H. Shmerling, MD

If you’ve ever tried to lose excess weight, you’ve probably gotten this advice: drink more water. Or perhaps it was more specific: drink a full glass of water before each meal.

The second suggestion seems like a reasonable idea, right? If you fill your stomach with water before eating, you’ll feel fuller and stop eating sooner. But did that work for you? Would drinking more water throughout the day work? Why do people say drinking water can help with weight loss — and what does the evidence show?

Stretching nerves, burning calories, and thirst versus hunger

Three top theories are:

Feel full, eat less. As noted, filling up on water before meals has intuitive appeal. Your stomach has nerves that sense stretch and send signals to the brain that it’s time to stop eating. Presumably, drinking before a meal could send similar signals.

• The evidence: Some small, short-term studies support this idea. For example, older study subjects who drank a full glass of water before meals tended to eat less than those who didn’t. Another study found that people following a low-calorie diet who drank extra water before meals had less appetite and more weight loss over 12 weeks than those on a similar diet without the extra water. But neither study assessed the impact of drinking extra water on long-term weight loss.

Burning off calories. The water we drink must be heated up to body temperature, a process requiring the body to expend energy. The energy spent on this — called thermogenesis — could offset calories from meals.

• The evidence: Though older studies provided some support for this explanation, more recent studies found no evidence that drinking water burned off many calories. That calls the thermogenesis explanation for water-induced weight loss into question.

You’re not hungry, you’re thirsty. This explanation suggests that sometimes we head to the kitchen for something to eat when we’re actually thirsty rather than hungry. If that’s the case, drinking calorie-free water can save us from consuming unnecessary calories — and that could promote weight loss.

• The evidence: The regulation of thirst and hunger is complex and varies over a person’s lifespan. For example, thirst may be dulled in older adults. But I could find no convincing studies in humans supporting the notion that people who are thirsty misinterpret the sensation for hunger, or that this is why drinking water might help with weight loss.

Exercise booster, no-cal substitution, and burning fat demands water

Being well-hydrated improves exercise capacity and thus weight loss. Muscle fatigue, cramping, and heat exhaustion can all be brought on by dehydration. That’s why extra hydration before exercise may be recommended, especially for elite athletes exercising in warm environments.

• The evidence: For most people, hydrating before exercises seems unnecessary, and I could find no studies specifically examining the role of hydration to exercise-related weight loss.

Swapping out high calorie drinks with water. Yes, if you usually drink high-calorie beverages (such as sweetened sodas, fruit juice, or alcohol), consistently replacing them with water can aid weight loss over time.

• The evidence: A dramatic reduction in calorie intake by substituting water for higher-calorie beverages could certainly lead to long-term weight loss. While it’s hard to design a study to prove this, indirect evidence suggests a link between substituting water for high-cal beverages and weight loss. Even so, just as calorie-restricting diets are hard to stick with over the long term, following a water-only plan may be easier said than done.

Burning fat requires water. Dehydration impairs the body’s ability to break down fat for fuel. So, perhaps drinking more water will encourage fat breakdown and, eventually, weight loss.

• The evidence: Though some animal studies support the idea, I could find no compelling evidence from human studies that drinking extra water helps burn fat as a means to lose excess weight.

The bottom line

So, should you bump up hydration by drinking water before or during meals, or even at other times during the day?

Some evidence does suggest this might aid weight loss, at least for some people. But those studies are mostly small or short-term, or based on animal data. Even positive studies only found modest benefits.

That said, if you think it’s working for you, there’s little downside to drinking a bit more water, other than the challenge of trying to drink if you aren’t particularly thirsty. My take? Though plenty of people recommend this approach, it seems based on a theory that doesn’t hold water.

About the Author

Robert H. Shmerling, MD, Senior Faculty Editor, Harvard Health Publishing; Editorial Advisory Board Member, Harvard Health Publishing

Dr. Robert H. Shmerling is the former clinical chief of the division of rheumatology at Beth Israel Deaconess Medical Center (BIDMC), and is a current member of the corresponding faculty in medicine at Harvard Medical School. … See Full Bio View all posts by Robert H. Shmerling, MD