  Image segmentation has been increasingly applied in medical settings as
recent developments have skyrocketed the potential applications of deep
learning. Urology, specifically, is one field of medicine that is primed for
the adoption of a real-time image segmentation system with the long-term aim of
automating endoscopic stone treatment. In this project, we explored supervised
deep learning models to annotate kidney stones in surgical endoscopic video
feeds. In this paper, we describe how we built a dataset from the raw videos
and how we developed a pipeline to automate as much of the process as possible.
For the segmentation task, we adapted and analyzed three baseline deep learning
models -- U-Net, U-Net++, and DenseNet -- to predict annotations on the frames
of the endoscopic videos with the highest accuracy above 90\%. To show clinical
potential for real-time use, we also confirmed that our best trained model can
accurately annotate new videos at 30 frames per second. Our results demonstrate
that the proposed method justifies continued development and study of image
segmentation to annotate ureteroscopic video feeds.
