With an increasing number of consumers seeking health advice from the newly available ChatGPT, researchers are keen to evaluate the reliability and accuracy of the information provided by the artificial intelligence chatbot.
According to a recent study conducted by the University of Maryland School of Medicine (UMSOM), the answers generated by ChatGPT are mostly accurate and correct a significant portion of the time. However, there are instances where the information provided may be inaccurate or even fictitious.
The results of the study, which was focused on evaluating the accuracy of ChatGPT’s responses to questions about breast cancer screening, were recently published in the Radiology journal. In February 2023, the researchers at UMSOM developed a list of 25 questions and submitted each one to the chatbot three times to examine the range of responses generated. A team of three radiologists who specialized in mammography then evaluated the responses, finding that 22 out of the 25 questions received appropriate answers. However, one of the answers was based on outdated information, and two other questions had inconsistent responses that varied significantly with each repeated inquiry.
They “found ChatGPT answered questions correctly about 88 percent of the time, which is pretty amazing,” remarks corresponding author Paul Yi.
“It also has the added benefit of summarizing information into an easily digestible form for consumers to easily understand.”
While ChatGPT provided accurate responses to questions related to the symptoms of breast cancer, those at risk, and recommendations regarding the cost, age, and frequency of mammograms, its responses are not as detailed as what one would typically obtain from a Google search.
“ChatGPT provided only one set of recommendations on breast cancer screening, issued from the American Cancer Society,” points out lead author Hana Haver, “but did not mention differing recommendations put out by the Centers for Disease Control and Prevention (CDC) or the US Preventative Services Task Force (USPSTF).”
The researchers flagged one of ChatGPT’s responses as inappropriate, in which the chatbot provided an outdated recommendation regarding scheduling mammograms around COVID-19 vaccination. In February 2022, the advice to delay mammograms for four to six weeks after receiving the COVID-19 shot was revised, and the Centers for Disease Control and Prevention (CDC) now endorses the United States Preventive Services Task Force (USPSTF) guidelines, which do not recommend waiting. In addition, ChatGPT provided inconsistent responses to questions concerning an individual’s personal risk of developing breast cancer and where one could undergo mammography screening.
“We’ve seen in our experience that ChatGPT sometimes makes up fake journal articles or health consortiums to support its claims,” adds Dr. Yi. “Consumers should be aware that these are new, unproven technologies, and should still rely on their doctor, rather than ChatGPT, for advice.”
The researcher and his team are currently examining the performance of ChatGPT in providing lung cancer screening recommendations and exploring avenues to enhance the chatbot’s recommendations to make them more precise and comprehensive, while also ensuring that they are accessible to individuals with limited education.
“With the rapid evolution of ChatGPT and other large language models,” adds Mark T. Gladwin, MD, Dean, University of Maryland School of Medicine, Vice President for Medical Affairs, University of Maryland, Baltimore, and the John Z. and Akiko K. Bowers Distinguished Professor, “we have a responsibility as a medical community to evaluate these technologies and protect our patients from potential harm that may come from incorrect screening recommendations or outdated preventive health strategies.”
Image Credit: Getty