  We derive the fast convergence rates of a deep neural network (DNN)
classifier with the rectified linear unit (ReLU) activation function learned
using the hinge loss. We consider three cases for a true model: (1) a smooth
decision boundary, (2) smooth conditional class probability, and (3) the margin
condition (i.e., the probability of inputs near the decision boundary is
small). We show that the DNN classifier learned using the hinge loss achieves
fast rate convergences for all three cases provided that the architecture
(i.e., the number of layers, number of nodes and sparsity). is carefully
selected. An important implication is that DNN architectures are very flexible
for use in various cases without much modification. In addition, we consider a
DNN classifier learned by minimizing the cross-entropy, and show that the DNN
classifier achieves a fast convergence rate under the condition that the
conditional class probabilities of most data are sufficiently close to either 1
or zero. This assumption is not unusual for image recognition because human
beings are extremely good at recognizing most images. To confirm our
theoretical explanation, we present the results of a small numerical study
conducted to compare the hinge loss and cross-entropy.
