  This paper aims to theoretically analyze the complexity of feature
transformations encoded in piecewise linear DNNs with ReLU layers. We propose
metrics to measure three types of complexities of transformations based on the
information theory. We further discover and prove the strong correlation
between the complexity and the disentanglement of transformations. Based on the
proposed metrics, we analyze two typical phenomena of the change of the
transformation complexity during the training process, and explore the ceiling
of a DNN's complexity. The proposed metrics can also be used as a loss to learn
a DNN with the minimum complexity, which also controls the over-fitting level
of the DNN and influences adversarial robustness, adversarial transferability,
and knowledge consistency. Comprehensive comparative studies have provided new
perspectives to understand the DNN.
