  Phase recognition in surgical videos is crucial for enhancing computer-aided
surgical systems as it enables automated understanding of sequential procedural
stages. Existing methods often rely on fixed temporal windows for video
analysis to identify dynamic surgical phases. Thus, they struggle to
simultaneously capture short-, mid-, and long-term information necessary to
fully understand complex surgical procedures. To address these issues, we
propose Multi-Scale Transformers for Surgical Phase Recognition (MuST), a novel
Transformer-based approach that combines a Multi-Term Frame encoder with a
Temporal Consistency Module to capture information across multiple temporal
scales of a surgical video. Our Multi-Term Frame Encoder computes
interdependencies across a hierarchy of temporal scales by sampling sequences
at increasing strides around the frame of interest. Furthermore, we employ a
long-term Transformer encoder over the frame embeddings to further enhance
long-term reasoning. MuST achieves higher performance than previous
state-of-the-art methods on three different public benchmarks.
