  Most existing deep neural networks are static, which means they can only do
inference at a fixed complexity. But the resource budget can vary substantially
across different devices. Even on a single device, the affordable budget can
change with different scenarios, and repeatedly training networks for each
required budget would be incredibly expensive. Therefore, in this work, we
propose a general method called MutualNet to train a single network that can
run at a diverse set of resource constraints. Our method trains a cohort of
model configurations with various network widths and input resolutions. This
mutual learning scheme not only allows the model to run at different
width-resolution configurations but also transfers the unique knowledge among
these configurations, helping the model to learn stronger representations
overall. MutualNet is a general training methodology that can be applied to
various network structures (e.g., 2D networks: MobileNets, ResNet, 3D networks:
SlowFast, X3D) and various tasks (e.g., image classification, object detection,
segmentation, and action recognition), and is demonstrated to achieve
consistent improvements on a variety of datasets. Since we only train the model
once, it also greatly reduces the training cost compared to independently
training several models. Surprisingly, MutualNet can also be used to
significantly boost the performance of a single network, if dynamic resource
constraint is not a concern. In summary, MutualNet is a unified method for both
static and adaptive, 2D and 3D networks. Codes and pre-trained models are
available at \url{https://github.com/taoyang1122/MutualNet}.
