Introduction: Visualization and image quality during ureteroscopy is essential for stone tracking during fragmentation. However, this can impacted by blood and debris intraoperatively. Novel applications of computer vision methods may augment endoscopic visualization and improve stone tracking. We sought to use computer vision techniques to automatically segment kidney stones in real-time during ureteroscopy.
Methods: Twenty separate videos of ureteroscopy were collected with digital ureteroscopes (Karl Storz Flex Xc). Frames from each video were extracted at 20 frames per second (fps). The training data was manually annotated to identify the stone in each frame. Three established image segmentation computer vision models, U-Net, U-Net++, and DenseNet, were trained using the annotated frames. Different iterations of each model were compared via hyperparameter tuning for optimization. Eighty percent of the data was used to train the models with 10% being used for validation and 10% being used for testing. Outcomes included Dice similarity coefficient (DSC), accuracy (per pixel), and area under the receiver operating characteristic curve (ROC-AUC).
Results: Mean duration of video was 22 seconds (SD ± 13s). The U-Net++ and U-Net models demonstrated similarly strong performances for stone segmentation with DSCs of 0.92 and 0.91, accuracies of 0.95 and 0.96, and ROC-AUC scores of 0.97 and 0.97, respectively. DenseNet had lower performance comparatively with a DSC of 0.73, accuracy of 0.87, and ROC-AUC of 0.91 (Fig. 1). Additionally, we implemented a working system for the processing of real-time video feeds with overlayed model predictions (Fig. 2). The models were able to annotate new videos at 30 fps and maintain accuracy.
Conclusions: Computer vision models demonstrate strong performance for automatic stone segmentation during ureteroscopy. Annotating new videos at 30 fps demonstrates the feasibility of their application real-time during surgery. With further optimization, these models could augment ureteroscopic vision and improve stone treatment.
.jpg)
.jpg)
