Introduction: The assessment of surgical competency is essential for clinical training and safety. Currently, there are no validated, objective, real-time tools for evaluating expertise during endoscopic surgery. We sought to apply validated, automated kidney stone segmentation software to distinguish expert and trainee surgeons during flexible ureteroscopy (fURS). Methods: Forty-six separate videos of fURS were prospectively recorded. Surgeons were categorized as ‘expert’ (n=2, fellowship trained endourologist, case volume of >100 fURS per year) or trainee (n=2, resident, <100 fURS per year), performing two defined tasks: stone localization, and holmium laser ablation (dusting). Surgeons were randomly assigned to one of these tasks per case. Stone localization was standardized as evaluation of the entire collecting system. For the localization task, we analyzed first 5 seconds of video after stone identification. For the laser ablation task, we analyzed the first 20 seconds of stone ablation. All videos were visually validated for quality and frames extracted at 30fps. Frames were analyzed by previously validated automated stone segmentation models. We performed a pixel-based analysis, evaluating percent occupancy of stone for each frame and compared differences between trainees and experts for each task. Results: Of 46 videos (N=14299 frames), 28 were evaluated for the localization task (14 trainee, 14 expert) and 18 for the laser ablation task (9 novice, 9 expert). The percentage of frames without stones identified was higher in trainees compared to experts for both localization (25% vs. 5%, p<0.01) and laser ablation (16% vs. 8%, p<0.01), indicating more frequent loss of stone visualization by trainees. Stones occupied more of each frame for trainees compared to experts for both localization (18% vs. 11% , p<0.01) and laser ablation (20% vs. 16%, p<0.01), suggesting closer stone visualization during stone treatment by trainees (Fig. 1). There was greater variation in frame-to-frame stone occupancy for trainees compared to experts during stone localization (2.9% vs 1.5%, p<0.01). Conclusions: Objective and automated computer vision-mediated analysis can distinguish surgical experience between experts and trainee surgeons performing fURS for kidney stones. SOURCE OF Funding: Endourologic Society Grant Funding
