  Reconstructing the scene of robotic surgery from the stereo endoscopic video
is an important and promising topic in surgical data science, which potentially
supports many applications such as surgical visual perception, robotic surgery
education and intra-operative context awareness. However, current methods are
mostly restricted to reconstructing static anatomy assuming no tissue
deformation, tool occlusion and de-occlusion, and camera movement. However,
these assumptions are not always satisfied in minimal invasive robotic
surgeries. In this work, we present an efficient reconstruction pipeline for
highly dynamic surgical scenes that runs at 28 fps. Specifically, we design a
transformer-based stereoscopic depth perception for efficient depth estimation
and a light-weight tool segmentor to handle tool occlusion. After that, a
dynamic reconstruction algorithm which can estimate the tissue deformation and
camera movement, and aggregate the information over time is proposed for
surgical scene reconstruction. We evaluate the proposed pipeline on two
datasets, the public Hamlyn Centre Endoscopic Video Dataset and our in-house
DaVinci robotic surgery dataset. The results demonstrate that our method can
recover the scene obstructed by the surgical tool and handle the movement of
camera in realistic surgical scenarios effectively at real-time speed.
