  We present a formulation of deep learning that aims at producing a large
margin classifier. The notion of margin, minimum distance to a decision
boundary, has served as the foundation of several theoretically profound and
empirically successful results for both classification and regression tasks.
However, most large margin algorithms are applicable only to shallow models
with a preset feature representation; and conventional margin methods for
neural networks only enforce margin at the output layer. Such methods are
therefore not well suited for deep networks.
  In this work, we propose a novel loss function to impose a margin on any
chosen set of layers of a deep network (including input and hidden layers). Our
formulation allows choosing any norm on the metric measuring the margin. We
demonstrate that the decision boundary obtained by our loss has nice properties
compared to standard classification loss functions. Specifically, we show
improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on
multiple tasks: generalization from small training sets, corrupted labels, and
robustness against adversarial perturbations. The resulting loss is general and
complementary to existing data augmentation (such as random/adversarial input
transform) and regularization techniques (such as weight decay, dropout, and
batch norm).
