  Recognizing various surgical tools, actions and phases from surgery videos is
an important problem in computer vision with exciting clinical applications.
Existing deep-learning-based methods for this problem either process each
surgical video as a series of independent images without considering their
dependence, or rely on complicated deep learning models to count for dependence
of video frames. In this study, we revealed from exploratory data analysis that
surgical videos enjoy relatively simple semantic structure, where the presence
of surgical phases and tools can be well modeled by a compact hidden Markov
model (HMM). Based on this observation, we propose an HMM-stabilized deep
learning method for tool presence detection. A wide range of experiments
confirm that the proposed approaches achieve better performance with lower
training and running costs, and support more flexible ways to construct and
utilize training data in scenarios where not all surgery videos of interest are
extensively labelled. These results suggest that popular deep learning
approaches with over-complicated model structures may suffer from inefficient
utilization of data, and integrating ingredients of deep learning and
statistical learning wisely may lead to more powerful algorithms that enjoy
competitive performance, transparent interpretation and convenient model
training simultaneously.
