Evolving Landscape of Large Language Models: An Evaluation of ChatGPT and Bard in Answering Patient Queries on Colonoscopy

The popularity and implementation of artificial intelligence (AI) enabled large language models (LLMs) powering chatbots are promising in healthcare, especially for patient queries and communication 1 .Lee et al 2 evaluated the performance of ChatGPT (version January 2023) for answering eight common patient questions related to colonoscopy and compared it to the responses available on hospital webpages.The study concluded that the performance of ChatGPT answers were similar to non-AI answers in ease of understanding and scientific adequacy.However, it must be noted that GPT-3.5 was used for evaluation.Since then, major updates have been released, specifically ChatGPT-4.0by OpenAI and updated Bard by Google in March 2023 and July 2023 3,4 , respectively.Due to the rapid evolution of LLMs, we compared the performance of three LLMs including ChatGPT-3.5,ChatGPT-4, and Bard (version July 2023, queries ran on July 17 th , 2023) to answer common patient inquiries related to colonoscopy.All three LLMs were given 47 questions twice and responses were recorded for both answers to access for consistency (Supplementary Table ).Responses were scored by two reviewers on a scale of 0-2 (completely correct=2, correct but incomplete=1, incorrect=0).Disagreement among scores between the two reviewers (2 fellows) was resolved by a third gastroenterologist and a single final rating was included in the results.Answers were considered "unreliable" if the two responses for the same query were inconsistent and had differences in meanings.Each response was evaluated in two simulated scenarios: firstly, as replies on a patient-oriented information platform, like informational websites run by hospitals; and secondly, as AI-crafted preliminary responses to electronic message queries sent by patients, intended for healthcare provider review.Among the three models, the performance of ChatGPT-4 was considered highest with 43 out of 47 (91.4%) responses graded as completely correct, while rest (8.6%) of the responses by ChatGPT-4 were graded as correct but incomplete.None of the responses were graded as incorrect for ChatGPT-4.For ChatGPT-3.5, of the 47 questions, only three (6.4%) were graded as completely correct, 40 (85.1%)we graded as correct but incomplete and 4 (8.5%) were considered as incorrect.Similar results were seen with Google's Bard LLM.Only seven (14.9%) J o u r n a l P r e -p r o o f responses were graded as completely correct, 30 (63.8%) were graded as correct but incomplete and 10 (21.3%) were considered as incorrect.None of the response were considered unreliable for ChatGPT-4 and ChatGPT-3.5,however two responses generated by Bard were considered unreliable.Although our study did not compare the performance of AI chatbots with human responses, our results highlight that various LLMs have differences in their performance as judged by expert gastroenterologists.As the landscape of LLMs has been evolving at an unprecedented pace, studies that utilize ChatGPT-3.5 may not be reliable and accurately represent the full potential and nuances of current LLM capabilities.The improvements in ChatGPT-4, although still with limitations, offers enhanced performance metrics and better understanding of context with a broader knowledge base and should be used in future studies.As LLMs continue to progress and become more prominent in medical use, there is a significant need for standardized regulation of performance metrics to ensure consistency, comparability, and reliability across different models 5 .For these models to be both clinically relevant and robust, it is vital that the medical research community be involved in their development, understanding their metrics and stay current with these rapid advancements.