  There has been a growing interest in using deep learning models for
processing long surgical videos, in order to automatically detect
clinical/operational activities and extract metrics that can enable workflow
efficiency tools and applications. However, training such models require vast
amounts of labeled data which is costly and not scalable. Recently,
self-supervised learning has been explored in computer vision community to
reduce the burden of the annotation cost. Masked autoencoders (MAE) got the
attention in self-supervised paradigm for Vision Transformers (ViTs) by
predicting the randomly masked regions given the visible patches of an image or
a video clip, and have shown superior performance on benchmark datasets.
However, the application of MAE in surgical data remains unexplored. In this
paper, we first investigate whether MAE can learn transferrable representations
in surgical video domain. We propose SurgMAE, which is a novel architecture
with a masking strategy based on sampling high spatio-temporal tokens for MAE.
We provide an empirical study of SurgMAE on two large scale long surgical video
datasets, and find that our method outperforms several baselines in low data
regime. We conduct extensive ablation studies to show the efficacy of our
approach and also demonstrate it's superior performance on UCF-101 to prove
it's generalizability in non-surgical datasets as well.
