  Training competitive deep video models is an order of magnitude slower than
training their counterpart image models. Slow training causes long research
cycles, which hinders progress in video understanding research. Following
standard practice for training image models, video model training assumes a
fixed mini-batch shape: a specific number of clips, frames, and spatial size.
However, what is the optimal shape? High resolution models perform well, but
train slowly. Low resolution models train faster, but they are inaccurate.
Inspired by multigrid methods in numerical optimization, we propose to use
variable mini-batch shapes with different spatial-temporal resolutions that are
varied according to a schedule. The different shapes arise from resampling the
training data on multiple sampling grids. Training is accelerated by scaling up
the mini-batch size and learning rate when shrinking the other dimensions. We
empirically demonstrate a general and robust grid schedule that yields a
significant out-of-the-box training speedup without a loss in accuracy for
different models (I3D, non-local, SlowFast), datasets (Kinetics,
Something-Something, Charades), and training settings (with and without
pre-training, 128 GPUs or 1 GPU). As an illustrative example, the proposed
multigrid method trains a ResNet-50 SlowFast network 4.5x faster (wall-clock
time, same hardware) while also improving accuracy (+0.8% absolute) on
Kinetics-400 compared to the baseline training method. Code is available
online.
