  Automated video-based assessment of surgical skills is a promising task in
assisting young surgical trainees, especially in poor-resource areas. Existing
works often resort to a CNN-LSTM joint framework that models long-term
relationships by LSTMs on spatially pooled short-term CNN features. However,
this practice would inevitably neglect the difference among semantic concepts
such as tools, tissues, and background in the spatial dimension, impeding the
subsequent temporal relationship modeling. In this paper, we propose a novel
skill assessment framework, Video Semantic Aggregation (ViSA), which discovers
different semantic parts and aggregates them across spatiotemporal dimensions.
The explicit discovery of semantic parts provides an explanatory visualization
that helps understand the neural network's decisions. It also enables us to
further incorporate auxiliary information such as the kinematic data to improve
representation learning and performance. The experiments on two datasets show
the competitiveness of ViSA compared to state-of-the-art methods. Source code
is available at: bit.ly/MICCAI2022ViSA.
