  Deep learning has achieved many breakthroughs in modern classification tasks.
Numerous architectures have been proposed for different data structures but
when it comes to the loss function, the cross-entropy loss is the predominant
choice. Recently, several alternative losses have seen revived interests for
deep classifiers. In particular, empirical evidence seems to promote square
loss but a theoretical justification is still lacking. In this work, we
contribute to the theoretical understanding of square loss in classification by
systematically investigating how it performs for overparametrized neural
networks in the neural tangent kernel (NTK) regime. Interesting properties
regarding the generalization error, robustness, and calibration error are
revealed. We consider two cases, according to whether classes are separable or
not. In the general non-separable case, fast convergence rate is established
for both misclassification rate and calibration error. When classes are
separable, the misclassification rate improves to be exponentially fast.
Further, the resulting margin is proven to be lower bounded away from zero,
providing theoretical guarantees for robustness. We expect our findings to hold
beyond the NTK regime and translate to practical settings. To this end, we
conduct extensive empirical studies on practical neural networks, demonstrating
the effectiveness of square loss in both synthetic low-dimensional data and
real image data. Comparing to cross-entropy, square loss has comparable
generalization error but noticeable advantages in robustness and model
calibration.
