Americans ask AI for health care. Hospitals think the answer is more chatbots.

Clear concerns

With so many Americans using artificial intelligence (AI) to fill health care gaps, there are now mounting cautionary tales and horror stories. The examples highlight pitfalls in both what the large language models (LLMs) are asked and what information they’re hoovering up.

A more cautious tool

Beyond PatientGPT, several health systems are moving forward with their own chatbots. Hartford HealthCare and K Health’s PatientGPT was rolled out as a beta version to select patients last month, and the company is planning to expand the rollout to tens of thousands more this week, according to Stat.

Hartford posted a pre-print study (not peer-reviewed) involving 75 participants that suggested its iterative stress testing improved its failure rate in “high risk” scenarios over time. The testing dropped the failure rate from 30 percent to 8.5 percent. But what that means for real-life settings is unclear—as is how bad the 8.5 percent failures might be.

According to Stat, PatientGPT works in two modes: a generic medical question-and-answer mode that may incorporate information about the patient, or a “medical intake” mode, in which a patient starts providing symptom information and the chatbot gets less chatty and starts going through clinical flowcharts. After the AI agent collects enough information in intake mode, it will provide a next step, including setting up a follow-up appointment with primary care or seeking urgent or emergency care. If the latter is recommended, the chatbot stops responding to further questions.

Hartford said it will continue to monitor the chatbot’s performance amid the larger rollout. In piloting, Hartford was monitoring every interaction. But now the system will drop down to having human reviews of just 20 interactions a day while a separate AI agent monitors the rest. They’ll also do batch studies of every 1,000 conversations.

“We’re on a mission to be the most consumer centric health system in the country,” Jeff Flaks, president and CEO of Hartford HealthCare, said last month. “So much of healthcare has traditionally been organized around the provider, but it’s clear we have to meet people where they are and where they desire to be met. With PatientGPT we are introducing a new tool that supports your health and provides access to a 24/7 care team, while protecting the human relationships at the heart of care.”

Key context

To consider AI’s potential role, it’s useful to consider the wider context of US health care. America is one of the wealthiest countries in the world, but its health care system consistently and significantly underperforms compared with those of other high-income countries. Americans have lower life expectancy, more avoidable deaths, higher rates of maternal and infant deaths, and higher rates of obesity and chronic conditions. America has less access to care and worse health outcomes. The US is an outlier in not providing universal care. A 2023 report found that nearly a third of Americans—more than 100 million people—don’t have a primary care provider.

Rollouts underway

Several health systems are slowly rolling out their own chatbots. Hartford HealthCare and K Health’s PatientGPT was rolled out as a beta version to select patients last month, and the company is planning to expand the rollout to tens of thousands more this week, according to Stat.

Epic, the electronic health records behemoth behind MyChart, is also rolling out its own AI chat assistant called Emmie. Several health systems are slowly rolling Emmie out to users through the online portal, including California-based Sutter Health and Indiana-based Reid Health.

Key Takeaways

• The rollouts of AI chatbots in US healthcare systems are raising immediate questions about their safety and efficacy.
• While some health systems view these chatbots as a convenient way to meet patients where they are, others are more cautious and are implementing monitoring and review processes.
• The potential benefits of AI chatbots for patient care remain largely hypothetical at this stage.
• There is concern about the quality of medical information that LLMs may pull in, with some examples of fake or misleading information being shared by these models.
• Hospitals are seeking to use AI as a tool to improve access and convenience for patients, but also need to ensure they do so in a way that does not compromise patient safety or privacy.