  In machine learning, there is a fundamental trade-off between ease of
optimization and expressive power. Neural Networks, in particular, have
enormous expressive power and yet are notoriously challenging to train. The
nature of that optimization challenge changes over the course of learning.
Traditionally in deep learning, one makes a static trade-off between the needs
of early and late optimization. In this paper, we investigate a novel
framework, GradNets, for dynamically adapting architectures during training to
get the benefits of both. For example, we can gradually transition from linear
to non-linear networks, deterministic to stochastic computation, shallow to
deep architectures, or even simple downsampling to fully differentiable
attention mechanisms. Benefits include increased accuracy, easier convergence
with more complex architectures, solutions to test-time execution of batch
normalization, and the ability to train networks of up to 200 layers.
