  Automatic surgical phase recognition is a challenging and crucial task with
the potential to improve patient safety and become an integral part of
intra-operative decision-support systems. In this paper, we propose, for the
first time in workflow analysis, a Multi-Stage Temporal Convolutional Network
(MS-TCN) that performs hierarchical prediction refinement for surgical phase
recognition. Causal, dilated convolutions allow for a large receptive field and
online inference with smooth predictions even during ambiguous transitions. Our
method is thoroughly evaluated on two datasets of laparoscopic cholecystectomy
videos with and without the use of additional surgical tool information.
Outperforming various state-of-the-art LSTM approaches, we verify the
suitability of the proposed causal MS-TCN for surgical phase recognition.
