  Training of deep models for classification tasks is hindered by local minima
problems and vanishing gradients, while unsupervised layer-wise pretraining
does not exploit information from class labels. Here, we propose a new
regularization technique, called diversifying regularization (DR), which
applies a penalty on hidden units at any layer if they obtain similar features
for different types of data. For generative models, DR is defined as divergence
over the variational posteriori distributions and included in the maximum
likelihood estimation as a prior. Thus, DR includes class label information for
greedy pretraining of deep belief networks which result in a better weight
initialization for fine-tuning methods. On the other hand, for discriminative
training of deep neural networks, DR is defined as a distance over the features
and included in the learning objective. With our experimental tests, we show
that DR can help the backpropagation to cope with vanishing gradient problems
and to provide faster convergence and smaller generalization errors.
