Assessment of Challenging Oncologic Cases: A Comparative Analysis Between ChatGPT, Gemini, and a Multidisciplinary Tumor Board
ABSTRACT
Introduction
Since its introduction in 2022, public‐access conversational AI, exemplified by ChatGPT and Gemini, has been increasingly utilized in medical decision‐making, though its impact is questionable. This study aims to evaluate its efficacy in assessing complex oncologic cases compared to a multidisciplinary tumor board (MTB) comprising experts from various specialties.
Methods
A 2‐year retrospective analysis was conducted on 98 oncological cases at a reference medical center in Mexico City. A MTB comprising surgical oncologists, medical oncologists, radio‐oncologists, pathologists, among others, reviewed and discussed each case to determine management strategies. We evaluated four key decision points, dichotomized as either affirmative or negative: the need for new imaging studies, radiation therapy, chemotherapy, and surgery. Comprehensive medical documentation accompanied each case. We then compared AI's decisions with those of the MTB using the same criteria and conducted a Cohen's Kappa test to assess agreement.
Results
Agreement between ChatGPT (4o) and Gemini (1.5 Flash), and the MTB ranged from none to slight for additional imaging studies (Gemini: κ = 0.100, p = 0.087; ChatGPT 4o: κ = 0.024, p = 0.592) and chemotherapy (Gemini: κ = 0.089, p = 0.316; ChatGPT 4o: κ = 0.336, p = 0.001). Moderate agreement was observed for decisions regarding surgery (Gemini: κ = 0.194, p = 0.046; ChatGPT 4o: κ = 0.467, p = < 0.001) and radiotherapy (Gemini: κ = 0.214, p = 0.012; ChatGPT 4o: κ = 0.525, p = < 0.001).
Conclusions
Both ChatGPT and Gemini showed moderate agreement with the multidisciplinary tumor board on decisions regarding surgery and radiotherapy. ChatGPT also showed moderate agreement in chemotherapy, but further assessment is needed for other interventions. ChatGPT proved to be superior to Gemini in most key points. The potential of these public access AI in oncology warrants continued exploration to refine its utility in clinical practice.
Top-30
Journals
|
1
|
|
|
Journal of Surgical Oncology
1 publication, 12.5%
|
|
|
Melanoma Research
1 publication, 12.5%
|
|
|
Journal of Cancer Research and Clinical Oncology
1 publication, 12.5%
|
|
|
Diagnostics
1 publication, 12.5%
|
|
|
Scientific Reports
1 publication, 12.5%
|
|
|
Journal of Medical Systems
1 publication, 12.5%
|
|
|
European urology oncology
1 publication, 12.5%
|
|
|
Vascular Medicine
1 publication, 12.5%
|
|
|
1
|
Publishers
|
1
2
3
|
|
|
Springer Nature
3 publications, 37.5%
|
|
|
Wiley
1 publication, 12.5%
|
|
|
Ovid Technologies (Wolters Kluwer Health)
1 publication, 12.5%
|
|
|
MDPI
1 publication, 12.5%
|
|
|
Elsevier
1 publication, 12.5%
|
|
|
SAGE
1 publication, 12.5%
|
|
|
1
2
3
|
- We do not take into account publications without a DOI.
- Statistics recalculated weekly.