  Recognition of surgical gesture is crucial for surgical skill assessment and
efficient surgery training. Prior works on this task are based on either
variant graphical models such as HMMs and CRFs, or deep learning models such as
Recurrent Neural Networks and Temporal Convolutional Networks. Most of the
current approaches usually suffer from over-segmentation and therefore low
segment-level edit scores. In contrast, we present an essentially different
methodology by modeling the task as a sequential decision-making process. An
intelligent agent is trained using reinforcement learning with hierarchical
features from a deep model. Temporal consistency is integrated into our action
design and reward mechanism to reduce over-segmentation errors. Experiments on
JIGSAWS dataset demonstrate that the proposed method performs better than
state-of-the-art methods in terms of the edit score and on par in frame-wise
accuracy. Our code will be released later.
