  Analyzing surgical workflow is crucial for surgical assistance robots to
understand surgeries. With the understanding of the complete surgical workflow,
the robots are able to assist the surgeons in intra-operative events, such as
by giving a warning when the surgeon is entering specific keys or high-risk
phases. Deep learning techniques have recently been widely applied to
recognizing surgical workflows. Many of the existing temporal neural network
models are limited in their capability to handle long-term dependencies in the
data, instead, relying upon the strong performance of the underlying per-frame
visual models. We propose a new temporal network structure that leverages
task-specific network representation to collect long-term sufficient statistics
that are propagated by a sufficient statistics model (SSM). We implement our
approach within an LSTM backbone for the task of surgical phase recognition and
explore several choices for propagated statistics. We demonstrate superior
results over existing and novel state-of-the-art segmentation techniques on two
laparoscopic cholecystectomy datasets: the publicly available Cholec80 dataset
and MGH100, a novel dataset with more challenging and clinically meaningful
segment labels.
