  In this paper, the robustness and accuracy of the deep neural network (DNN)
was enhanced by introducing the $L_{2,\infty}$ normalization of the weight
matrices of the DNN with Relu as the activation function. It is proved that the
$L_{2,\infty}$ normalization leads to large dihedral angles between two
adjacent faces of the polyhedron graph of the DNN function and hence smoother
DNN functions, which reduces over-fitting. A measure is proposed for the
robustness of a classification DNN, which is the average radius of the maximal
robust spheres with the sample data as centers. A lower bound for the
robustness measure is given in terms of the $L_{2,\infty}$ norm. Finally, an
upper bound for the Rademacher complexity of DNN with $L_{2,\infty}$
normalization is given. An algorithm is given to train a DNN with the
$L_{2,\infty}$ normalization and experimental results are used to show that the
$L_{2,\infty}$ normalization is effective to improve the robustness and
accuracy.
