  Analyzing the behavior of ReLU neural networks often hinges on understanding
the relationships between their parameters and the functions they implement.
This paper proves a new bound on function distances in terms of the so-called
path-metrics of the parameters. Since this bound is intrinsically invariant
with respect to the rescaling symmetries of the networks, it sharpens
previously known bounds. It is also, to the best of our knowledge, the first
bound of its kind that is broadly applicable to modern networks such as
ResNets, VGGs, U-nets, and many more. In contexts such as network pruning and
quantization, the proposed path-metrics can be efficiently computed using only
two forward passes. Besides its intrinsic theoretical interest, the bound
yields not only novel theoretical generalization bounds, but also a promising
proof of concept for rescaling-invariant pruning.
