  One of the main challenges for feature representation in deep learning-based
classification is the design of appropriate loss functions that exhibit strong
discriminative power. The classical softmax loss does not explicitly encourage
discriminative learning of features. A popular direction of research is to
incorporate margins in well-established losses in order to enforce extra
intra-class compactness and inter-class separability, which, however, were
developed through heuristic means, as opposed to rigorous mathematical
principles. In this work, we attempt to address this limitation by formulating
the principled optimization objective as learning towards the largest margins.
Specifically, we firstly define the class margin as the measure of inter-class
separability, and the sample margin as the measure of intra-class compactness.
Accordingly, to encourage discriminative representation of features, the loss
function should promote the largest possible margins for both classes and
samples. Furthermore, we derive a generalized margin softmax loss to draw
general conclusions for the existing margin-based losses. Not only does this
principled framework offer new perspectives to understand and interpret
existing margin-based losses, but it also provides new insights that can guide
the design of new tools, including sample margin regularization and largest
margin softmax loss for the class-balanced case, and zero-centroid
regularization for the class-imbalanced case. Experimental results demonstrate
the effectiveness of our strategy on a variety of tasks, including visual
classification, imbalanced classification, person re-identification, and face
verification.
