  Automatic surgical gesture recognition is a prerequisite of intra-operative
computer assistance and objective surgical skill assessment. Prior works either
require additional sensors to collect kinematics data or have limitations on
capturing temporal information from long and untrimmed surgical videos. To
tackle these challenges, we propose a novel temporal convolutional architecture
to automatically detect and segment surgical gestures with corresponding
boundaries only using RGB videos. We devise our method with a symmetric
dilation structure bridged by a self-attention module to encode and decode the
long-term temporal patterns and establish the frame-to-frame relationship
accordingly. We validate the effectiveness of our approach on a fundamental
robotic suturing task from the JIGSAWS dataset. The experiment results
demonstrate the ability of our method on capturing long-term frame
dependencies, which largely outperform the state-of-the-art methods on the
frame-wise accuracy up to ~6 points and the F1@50 score ~6 points.
