  To assist surgeons in the operating theatre, surgical phase recognition is
critical for developing computer-assisted surgical systems, which requires
comprehensive understanding of surgical videos. Although existing studies made
great progress, there are still two significant limitations worthy of
improvement. First, due to the compromise of resource consumption, frame-wise
visual features are extracted by 2D networks and disregard spatial and temporal
knowledge of surgical actions, which hinders subsequent inter-frame modeling
for phase prediction. Second, these works simply utilize ordinary
classification loss with one-hot phase labels to optimize the phase
predictions, and cannot fully explore surgical videos under inadequate
supervision. To overcome these two limitations, we propose a Surgical Temporal
Action-aware Network with sequence Regularization, named STAR-Net, to recognize
surgical phases more accurately from input videos. Specifically, we propose an
efficient multi-scale surgical temporal action (MS-STA) module, which
integrates visual features with spatial and temporal knowledge of surgical
actions at the cost of 2D networks. Moreover, we devise the dual-classifier
sequence regularization (DSR) to facilitate the training of STAR-Net by the
sequence guidance of an auxiliary classifier with a smaller capacity. Our
STAR-Net with MS-STA and DSR can exploit visual features of surgical actions
with effective regularization, thereby leading to the superior performance of
surgical phase recognition. Extensive experiments on a large-scale gastrectomy
surgery dataset and the public Cholec80 benchmark prove that our STAR-Net
significantly outperforms state-of-the-arts of surgical phase recognition.
