  We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to
accelerate DNN training. Our algorithm uses large mini-batches to compute an
approximate solution quickly and then refines it by averaging the weights of
multiple models computed independently and in parallel. The resulting models
generalize equally well as those trained with small mini-batches but are
produced in a substantially shorter time. We demonstrate the reduction in
training time and the good generalization performance of the resulting models
on the computer vision datasets CIFAR10, CIFAR100, and ImageNet.
