  Larger networks generally have greater representational power at the cost of
increased computational complexity. Sparsifying such networks has been an
active area of research but has been generally limited to static regularization
or dynamic approaches using reinforcement learning. We explore a mixture of
experts (MoE) approach to deep dynamic routing, which activates certain experts
in the network on a per-example basis. Our novel DeepMoE architecture increases
the representational power of standard convolutional networks by adaptively
sparsifying and recalibrating channel-wise features in each convolutional
layer. We employ a multi-headed sparse gating network to determine the
selection and scaling of channels for each input, leveraging exponential
combinations of experts within a single convolutional network. Our proposed
architecture is evaluated on four benchmark datasets and tasks, and we show
that Deep-MoEs are able to achieve higher accuracy with lower computation than
standard convolutional networks.
