  Lipschitz-constrained neural networks have many applications in machine
learning. Since designing and training expressive Lipschitz-constrained
networks is very challenging, there is a need for improved methods and a better
theoretical understanding. Unfortunately, it turns out that ReLU networks have
provable disadvantages in this setting. Hence, we propose to use learnable
spline activation functions with at least 3 linear regions instead. We prove
that this choice is optimal among all component-wise $1$-Lipschitz activation
functions in the sense that no other weight constrained architecture can
approximate a larger class of functions. Additionally, this choice is at least
as expressive as the recently introduced non component-wise Groupsort
activation function for spectral-norm-constrained weights. Previously published
numerical results support our theoretical findings.
