  We develop a new method for regularising neural networks. We learn a
probability distribution over the activations of all layers of the model and
then insert imputed values into the network during training. We obtain a
posterior for an arbitrary subset of activations conditioned on the remainder.
This is a generalisation of data augmentation to the hidden layers of a
network, and a form of data-aware dropout. We demonstrate that our training
method leads to higher test accuracy and lower test-set cross-entropy for
neural networks trained on CIFAR-10 and SVHN compared to standard
regularisation baselines: our approach leads to networks with better calibrated
uncertainty over the class posteriors all the while delivering greater test-set
accuracy.
