  Surgical phase recognition is of particular interest to computer assisted
surgery systems, in which the goal is to predict what phase is occurring at
each frame for a surgery video. Networks with multi-stage architecture have
been widely applied in many computer vision tasks with rich patterns, where a
predictor stage first outputs initial predictions and an additional refinement
stage operates on the initial predictions to perform further refinement.
Existing works show that surgical video contents are well ordered and contain
rich temporal patterns, making the multi-stage architecture well suited for the
surgical phase recognition task. However, we observe that when simply applying
the multi-stage architecture to the surgical phase recognition task, the
end-to-end training manner will make the refinement ability fall short of its
wishes. To address the problem, we propose a new non end-to-end training
strategy and explore different designs of multi-stage architecture for surgical
phase recognition task. For the non end-to-end training strategy, the
refinement stage is trained separately with proposed two types of disturbed
sequences. Meanwhile, we evaluate three different choices of refinement models
to show that our analysis and solution are robust to the choices of specific
multi-stage models. We conduct experiments on two public benchmarks, the
M2CAI16 Workflow Challenge, and the Cholec80 dataset. Results show that
multi-stage architecture trained with our strategy largely boosts the
performance of the current state-of-the-art single-stage model. Code is
available at \url{https://github.com/ChinaYi/casual_tcn}.
