  There remains a list of unanswered research questions on deep learning (DL),
including the remarkable generalization power of overparametrized neural
networks, the efficient optimization performance despite the non-convexity, and
the mechanisms behind flat minima in generalization. In this paper, we adopt an
information-theoretic perspective to explore the theoretical foundations of
supervised classification using deep neural networks (DNNs). Our analysis
introduces the concepts of fitting error and model risk, which, together with
generalization error, constitute an upper bound on the expected risk. We
demonstrate that the generalization errors are bounded by the complexity,
influenced by both the smoothness of distribution and the sample size.
Consequently, task complexity serves as a reliable indicator of the dataset's
quality, guiding the setting of regularization hyperparameters. Furthermore,
the derived upper bound fitting error links the back-propagated gradient,
Neural Tangent Kernel (NTK), and the model's parameter count with the fitting
error. Utilizing the triangle inequality, we establish an upper bound on the
expected risk. This bound offers valuable insights into the effects of
overparameterization, non-convex optimization, and the flat minima in
DNNs.Finally, empirical verification confirms a significant positive
correlation between the derived theoretical bounds and the practical expected
risk, confirming the practical relevance of the theoretical findings.
