Introduction: To validate a fully automated artificial intelligence (AI) support system for urine cytology to detect histological high-grade urothelial carcinoma (HGUC). Methods: A total of 660 urine cytology slides were collected from the urine of consecutive patients just before bladder biopsy or transurethral resection of the bladder at 3 institutions (center 1, Kyoto Prefectural University of Medicine Hospital; center 2, Japanese Red Cross Kyoto Second Hospital; center 3, North Medical Center Kyoto Prefectural University of Medicine) between 2016 to 2022 (IRB ERB-C 1339-5, 1673-1, and S2020-60). Board-certified cytotechnologists and pathologist independently labeled each cell and classified each slide according to the Paris system (TPS). After excluding an unsatisfactory slide, cytology slides were digitized for image analysis. AI was developed with a deep learning method using 181 slides obtained in center 1. Urinary cells on a slide were automatically detected by AI, each cell was classified into benign vs malignant, the results of cell-level classifications were integrated, and AI classified each slide into negative (benign or atypical urothelial cells) vs positive (suspicious HGUC or HGUC). The model showing the highest accuracy in the training dataset was selected, and the slide classification performance was tested with 315 slides obtained from center 1 for internal validation and 163 slides obtained from centers 2 and 3 for external validation. The diagnostic performance to detect histological HGUC assessed with TPS and slide classification with AI were compared. The receiver operating characteristic (ROC) analysis was performed for the binary classification. Statistical significance defined as p<0.05. Results: For slide classification, the area under the ROC (AUC) of AI was 0.93 in internal validation and 0.84 in external validation. The accuracy was 85% in both internal and external datasets. For diagnostic performance to detect histological HGUC, AUC was 0.78 in internal validation and 0.73 in external validation. The accuracy/sensitivity/specificity with TPS was 68%/46%/89% in internal dataset and 66%/35%/91% in external dataset. When the sensitivity was matched with TPS, the accuracy of AI was 68% (p=1.0) and 66% (p=1.0) and specificity was 89% (p=0.6) and 91% (p=0.7) for internal and external validation, respectively. Conclusions: A fully automated AI system accurately classifies the digitized urine cytology slides and detects histological HGUC with comparative performance to TPS in both internal and external validation. SOURCE OF Funding: None.
