  We investigate the complexity of deep neural networks through the lens of
functional equivalence, which posits that different parameterizations can yield
the same network function. Leveraging the equivalence property, we present a
novel bound on the covering number for deep neural networks, which reveals that
the complexity of neural networks can be reduced. Additionally, we demonstrate
that functional equivalence benefits optimization, as overparameterized
networks tend to be easier to train since increasing network width leads to a
diminishing volume of the effective parameter space. These findings can offer
valuable insights into the phenomenon of overparameterization and have
implications for understanding generalization and optimization in deep
learning.
