  This paper reports a novel deep architecture referred to as Maxout network In
Network (MIN), which can enhance model discriminability and facilitate the
process of information abstraction within the receptive field. The proposed
network adopts the framework of the recently developed Network In Network
structure, which slides a universal approximator, multilayer perceptron (MLP)
with rectifier units, to exact features. Instead of MLP, we employ maxout MLP
to learn a variety of piecewise linear activation functions and to mediate the
problem of vanishing gradients that can occur when using rectifier units.
Moreover, batch normalization is applied to reduce the saturation of maxout
units by pre-conditioning the model and dropout is applied to prevent
overfitting. Finally, average pooling is used in all pooling layers to
regularize maxout MLP in order to facilitate information abstraction in every
receptive field while tolerating the change of object position. Because average
pooling preserves all features in the local patch, the proposed MIN model can
enforce the suppression of irrelevant information during training. Our
experiments demonstrated the state-of-the-art classification performance when
the MIN model was applied to MNIST, CIFAR-10, and CIFAR-100 datasets and
comparable performance for SVHN dataset.
