  Compressing deep neural networks while maintaining accuracy is important when
we want to deploy large, powerful models in production and/or edge devices. One
common technique used to achieve this goal is knowledge distillation.
Typically, the output of a static pre-defined teacher (a large base network) is
used as soft labels to train and transfer information to a student (or smaller)
network. In this paper, we introduce Adjoined Networks, or AN, a learning
paradigm that trains both the original base network and the smaller compressed
network together. In our training approach, the parameters of the smaller
network are shared across both the base and the compressed networks. Using our
training paradigm, we can simultaneously compress (the student network) and
regularize (the teacher network) any architecture. In this paper, we focus on
popular CNN-based architectures used for computer vision tasks. We conduct an
extensive experimental evaluation of our training paradigm on various
large-scale datasets. Using ResNet-50 as the base network, AN achieves 71.8%
top-1 accuracy with only 1.8M parameters and 1.6 GFLOPs on the ImageNet
data-set. We further propose Differentiable Adjoined Networks (DAN), a training
paradigm that augments AN by using neural architecture search to jointly learn
both the width and the weights for each layer of the smaller network. DAN
achieves ResNet-50 level accuracy on ImageNet with $3.8\times$ fewer parameters
and $2.2\times$ fewer FLOPs.
