ChatGPT in urogynecology, novel or not?

Thursday, October 5, 2023

3:30 PM – 4:30 PM US PST

Location: B117

Click to View Image
Image Label: Accuracy of ChatGPT in Generating Novel Systematic Review Ideas in Urogynecology

Scientific Table Presenter(s)

David M. Choueka, MD (he/him/his)

Resident
Department of Obstetrics and Gynecology, Lenox Hill Hospital, Zucker School of Medicine at Hofstra/Northwell
New York, New York, United States

Co-Author(s)

Alexandra Tabakin, MD

Fellow
Division of Female Pelvic Medicine and Reconstructive Surgery, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell
Great Neck, New York, United States
DS

Dara F. Shalom, MD, FACOG, FPMRS (she/her/hers)

Associate Chief of Urogynecology, Fellowship Program Director
Northwell Health
Great Neck, New York, United States

Disclosure(s):

David M. Choueka, MD: No financial relationships to disclose

Alexandra Tabakin, MD: No financial relationships to disclose

Dara F. Shalom, MD, FACOG, FPMRS: No relevant disclosure to display

Objectives: ChatGPT (Chat Generative Pre-trained Transformer) is an artificial intelligence (AI) chatbot that utilizes natural language processing which allows computers to interpret text and conversations. ChatGPT has been used for many novel applications in medicine, including research and scientific writing. Little is known about the effective utilization of ChatGPT as a research tool in identifying gaps in the urogynecology literature open for further exploration. Our objective was to evaluate the accuracy of ChatGPT in generating novel systematic reviews ideas in urogynecology.

Methods: In 3 unique prompts, we asked ChatGPT to generate 10 novel systematic review ideas that have not yet been published in 1) urogynecology, 2) on pessaries for pelvic organ prolapse (POP), and 3) on tension-free vaginal tape slings (TVTS). Systematic review novelty was assessed by cross-referencing Pubmed and SCOPUS to determine if any prior publications existed.

Results: ChatGPT proposed 30 total “novel” systematic review ideas, including 10 for each prompt. Overall, ChatGPT demonstrated 53% overall accuracy in developing novel systematic review ideas. When stratified by topic, we found that ChatGPT’s accuracy for generating novel research ideas was 50% for general urogynecology, 30% for TVTS, and 70% for pessaries for POP. For the 10 “novel ideas” suggested for general urogynecology, TVTS, and pessaries for POP, respectively, a total of 32, 11, and 9 systematic reviews had already been published. There were no publications of any type identified for 7 of the 30 “novel” ideas (1 general urogynecology, 3 TVTS, and 3 pessary for POP).

Conclusions: ChatGPT may be helpful in identifying opportunities for novel research for clinicians and researchers in urogynecology, but its accuracy is limited. For those using the current ChatGPT system to generate research ideas, we recommend that they cross-reference other literature databases to ensure originality. Ultimately, we encourage all urogynecologists to familiarize themselves with popular AI platforms, as AI is a rapidly evolving, dynamic technology which is set to massively impact healthcare.