  Gradient-based algorithms for training ResNets typically require a forward
pass of the input data, followed by back-propagating the objective gradient to
update parameters, which are time-consuming for deep ResNets. To break the
dependencies between modules in both the forward and backward modes,
auxiliary-variable methods such as the penalty and augmented Lagrangian (AL)
approaches have attracted much interest lately due to their ability to exploit
layer-wise parallelism. However, we observe that large communication overhead
and lacking data augmentation are two key challenges of these methods, which
may lead to low speedup ratio and accuracy drop across multiple compute
devices. Inspired by the optimal control formulation of ResNets, we propose a
novel serial-parallel hybrid training strategy to enable the use of data
augmentation, together with downsampling filters to reduce the communication
cost. The proposed strategy first trains the network parameters by solving a
succession of independent sub-problems in parallel and then corrects the
network parameters through a full serial forward-backward propagation of data.
Such a strategy can be applied to most of the existing layer-parallel training
methods using auxiliary variables. As an example, we validate the proposed
strategy using penalty and AL methods on ResNet and WideResNet across MNIST,
CIFAR-10 and CIFAR-100 datasets, achieving significant speedup over the
traditional layer-serial training methods while maintaining comparable
accuracy.
