  Deep learning systems have been reported to achieve state-of-the-art
performances in many applications, and a key is the existence of well trained
classifiers on benchmark datasets. As a main-stream loss function, the cross
entropy can easily lead us to find models which demonstrate severe overfitting
behavior. In this paper, we show that the existing cross entropy loss
minimization problem essentially learns the label conditional entropy (CE) of
the underlying data distribution of the dataset. However, the CE learned in
this way does not characterize well the information shared by the label and the
input. In this paper, we propose a mutual information learning framework where
we train deep neural network classifiers via learning the mutual information
between the label and the input. Theoretically, we give the population
classification error lower bound in terms of the mutual information. In
addition, we derive the mutual information lower and upper bounds for a
concrete binary classification data model in $\mathbb{R}^n$, and also the error
probability lower bound in this scenario. Empirically, we conduct extensive
experiments on several benchmark datasets to support our theory. The mutual
information learned classifiers (MILCs) achieve far better generalization
performances than the conditional entropy learned classifiers (CELCs) with an
improvement which can exceed more than 10\% in testing accuracy.
