  In this paper, we propose the nonlinearity generation method to speed up and
stabilize the training of deep convolutional neural networks. The proposed
method modifies a family of activation functions as nonlinearity generators
(NGs). NGs make the activation functions linear symmetric for their inputs to
lower model capacity, and automatically introduce nonlinearity to enhance the
capacity of the model during training. The proposed method can be considered an
unusual form of regularization: the model parameters are obtained by training a
relatively low-capacity model, that is relatively easy to optimize at the
beginning, with only a few iterations, and these parameters are reused for the
initialization of a higher-capacity model. We derive the upper and lower bounds
of variance of the weight variation, and show that the initial symmetric
structure of NGs helps stabilize training. We evaluate the proposed method on
different frameworks of convolutional neural networks over two object
recognition benchmark tasks (CIFAR-10 and CIFAR-100). Experimental results
showed that the proposed method allows us to (1) speed up the convergence of
training, (2) allow for less careful weight initialization, (3) improve or at
least maintain the performance of the model at negligible extra computational
cost, and (4) easily train a very deep model.
