  Dropout and other feature noising schemes have shown promising results in
controlling over-fitting by artificially corrupting the training data. Though
extensive theoretical and empirical studies have been performed for generalized
linear models, little work has been done for support vector machines (SVMs),
one of the most successful approaches for supervised learning. This paper
presents dropout training for both linear SVMs and the nonlinear extension with
latent representation learning. For linear SVMs, to deal with the intractable
expectation of the non-smooth hinge loss under corrupting distributions, we
develop an iteratively re-weighted least square (IRLS) algorithm by exploring
data augmentation techniques. Our algorithm iteratively minimizes the
expectation of a re-weighted least square problem, where the re-weights are
analytically updated. For nonlinear latent SVMs, we consider learning one layer
of latent representations in SVMs and extend the data augmentation technique in
conjunction with first-order Taylor-expansion to deal with the intractable
expected non-smooth hinge loss and the nonlinearity of latent representations.
Finally, we apply the similar data augmentation ideas to develop a new IRLS
algorithm for the expected logistic loss under corrupting distributions, and we
further develop a non-linear extension of logistic regression by incorporating
one layer of latent representations. Our algorithms offer insights on the
connection and difference between the hinge loss and logistic loss in dropout
training. Empirical results on several real datasets demonstrate the
effectiveness of dropout training on significantly boosting the classification
accuracy of both linear and nonlinear SVMs. In addition, the nonlinear SVMs
further improve the prediction performance on several image datasets.
