1777: A Symptom Checker App Is Better at Classifying Inflammatory Disease Than Physicians Who Are Presented Identical Information

markus Gräf¹, Johannes Knitza², Jan Leipe³, Martin Krusche⁴, Martin Welcker⁵, Sebastian Kuhn⁶, Johanna Mucke⁷, Axel Hueber⁸, Johannes Hornig⁹, Philipp Klemm¹⁰, stefan Kleinert¹¹, Peer Aries¹², Nicolas Vuillerme¹³, Arnd Kleyer², Georg Schett¹⁴ and Johanna Callhoff¹⁵, ¹Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany, ²Department of Internal Medicine 3 – Rheumatology and Immunology, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, 91054 Erlangen, Germany; Deutsches Zentrum Immuntherapie, Friedrich-Alexander-UniversityErlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany, ³Division of Rheumatology, Department of Medicine V, University Hospital Mannheim, Medical Faculty Mannheim of the University of Heidelberg, Heidelberg, Germany, ⁴University Hospital Hamburg-Eppendorf, Hamburg, Germany, ⁵GBR, Planegg, Germany, ⁶Department of Digital Medicine, Medical Faculty OWL, Bielefeld University, Bielefeld, Germany, ⁷Policlinic and Hiller Research Center, Heinrich-Heine-University Duesseldorf, Neuss, Germany, ⁸Paracelsus University Nürnberg, Erlangen, Germany, ⁹Rheumatologische Schwerpunktpraxis, Osnabrück, Germany, ¹⁰Campus Kerckhoff of Justus Liebig University Giessen, Bad Nauheim, Germany, ¹¹Praxisgemeinschaft Rheumatologie - Nephrologie (PGRN), Erlangen, Germany, ¹²Rheumatologie im Struenseehaus, Hamburg, Germany, ¹³Université Grenoble Alpes, AGEIS, Grenoble, France, ¹⁴Universitätsklinikum Erlangen, Erlangen, Germany, ¹⁵German Rheumatism Research Centre, Berlin, Germany

Background/Purpose: The purpose of this study was to compare the ability of an artificial intelligence-based symptom checker app (Ada) and physicians to determine the presence/absence of an inflammatory rheumatic disease (IRD), solely relying on basic health and symptom-related medical history.

Methods: In this survey study, German speaking physicians with prior rheumatology working experience were asked to determine IRD presence/absence and suggest diagnoses for twenty different vignettes. The vignettes included only basic health and symptom-related medical history from 13 non-IRD and 7 IRD-cases. The vignettes and diagnostic suggestions from Ada were retrieved from a previous study, where patients newly presenting to a rheumatology outpatient clinic used Ada for symptom assessment prior to the physician appointment. IRD detection rate and suggested diagnoses of participants and Ada were compared to the gold standard, the final physicians' diagnosis, reported on the discharge summary report from the local site.

Results: A total of 132 vignettes were completed by 33 physicians (mean age 39 (standard deviation (SD) 8.2) years, mean rheumatology working experience 8.8 (SD 7.1) years, median 6.0 years). Ada correctly detected IRD status more often compared to physicians (70% vs 53%, P = 0.002) according to top diagnosis (see table). Ada listed the correct diagnosis more often compared to physicians (54% vs 32%, P < 0.001) as top diagnosis as well as among the top 3 diagnoses 59% vs 42%, P < 0.001). Physicians were more likely to correctly detect IRD status with increasing work experience and self-perceived accuracy.

Conclusion: If rheumatologists have to classify patients as IRD or non-IRD based on health and symptom-related medical history, the AI-based Ada app has a higher detection rate of IRDs compared to rheumatologists. Symptom checkers could improve the current referral process by supporting the decision which patients most urgently need an appointment. This could optimize the usage of scarce rheumatology health care resources.
Accuracy, sensitivity, specifitiy, positive and negative predictive value of Ada and physicians for correct classification of inflammatory rheumatic diseases.
Disclosures: m. Gräf, None; J. Knitza, AbbVie, Novartis, ThermoFisher, UCB, ABATON, Sanofi, Medac, Lilly, BMS, Gilead, GSK, Werfen, Vila Health, Böhringer Ingelheim, Janssen, Galapagos, Chugai; J. Leipe, None; M. Krusche, None; M. Welcker, None; S. Kuhn, None; J. Mucke, None; A. Hueber, None; J. Hornig, None; P. Klemm, None; s. Kleinert, None; P. Aries, None; N. Vuillerme, None; A. Kleyer, None; G. Schett, None; J. Callhoff, None.