  We present Multiscale Vision Transformers (MViT) for video and image
recognition, by connecting the seminal idea of multiscale feature hierarchies
with transformer models. Multiscale Transformers have several
channel-resolution scale stages. Starting from the input resolution and a small
channel dimension, the stages hierarchically expand the channel capacity while
reducing the spatial resolution. This creates a multiscale pyramid of features
with early layers operating at high spatial resolution to model simple
low-level visual information, and deeper layers at spatially coarse, but
complex, high-dimensional features. We evaluate this fundamental architectural
prior for modeling the dense nature of visual signals for a variety of video
recognition tasks where it outperforms concurrent vision transformers that rely
on large scale external pre-training and are 5-10x more costly in computation
and parameters. We further remove the temporal dimension and apply our model
for image classification where it outperforms prior work on vision
transformers. Code is available at:
https://github.com/facebookresearch/SlowFast
