  Accurate and robust tracking and reconstruction of the surgical scene is a
critical enabling technology toward autonomous robotic surgery. Existing
algorithms for 3D perception in surgery mainly rely on geometric information,
while we propose to also leverage semantic information inferred from the
endoscopic video using image segmentation algorithms. In this paper, we present
a novel, comprehensive surgical perception framework, Semantic-SuPer, that
integrates geometric and semantic information to facilitate data association,
3D reconstruction, and tracking of endoscopic scenes, benefiting downstream
tasks like surgical navigation. The proposed framework is demonstrated on
challenging endoscopic data with deforming tissue, showing its advantages over
our baseline and several other state-of the-art approaches. Our code and
dataset are available at https://github.com/ucsdarclab/Python-SuPer.
