  Purpose: Automatic segmentation and classification of surgical activity is
crucial for providing advanced support in computer-assisted interventions and
autonomous functionalities in robot-assisted surgeries. Prior works have
focused on recognizing either coarse activities, such as phases, or
fine-grained activities, such as gestures. This work aims at jointly
recognizing two complementary levels of granularity directly from videos,
namely phases and steps. Method: We introduce two correlated surgical
activities, phases and steps, for the laparoscopic gastric bypass procedure. We
propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN)
along with a multi-task Convolutional Neural Network (CNN) training setup to
jointly predict the phases and steps and benefit from their complementarity to
better evaluate the execution of the procedure. We evaluate the proposed method
on a large video dataset consisting of 40 surgical procedures (Bypass40).
Results: We present experimental results from several baseline models for both
phase and step recognition on the Bypass40 dataset. The proposed MTMS-TCN
method outperforms in both phase and step recognition by 1-2% in accuracy,
precision and recall, compared to single-task methods. Furthermore, for step
recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM
based models in accuracy, precision, and recall. Conclusion: In this work, we
present a multi-task multi-stage temporal convolutional network for surgical
activity recognition, which shows improved results compared to single-task
models on the Bypass40 gastric bypass dataset with multi-level annotations. The
proposed method shows that the joint modeling of phases and steps is beneficial
to improve the overall recognition of each type of activity.
