  Dropout is a popular technique for regularizing artificial neural networks.
Dropout networks are generally trained by minibatch gradient descent with a
dropout mask turning off some of the units---a different pattern of dropout is
applied to every sample in the minibatch. We explore a very simple alternative
to the dropout mask. Instead of masking dropped out units by setting them to
zero, we perform matrix multiplication using a submatrix of the weight
matrix---unneeded hidden units are never calculated. Performing dropout
batchwise, so that one pattern of dropout is used for each sample in a
minibatch, we can substantially reduce training times. Batchwise dropout can be
used with fully-connected and convolutional neural networks.
