  We propose to optimize the activation functions of a deep neural network by
adding a corresponding functional regularization to the cost function. We
justify the use of a second-order total-variation criterion. This allows us to
derive a general representer theorem for deep neural networks that makes a
direct connection with splines and sparsity. Specifically, we show that the
optimal network configuration can be achieved with activation functions that
are nonuniform linear splines with adaptive knots. The bottom line is that the
action of each neuron is encoded by a spline whose parameters (including the
number of knots) are optimized during the training procedure. The scheme
results in a computational structure that is compatible with the existing
deep-ReLU, parametric ReLU, APL (adaptive piecewise-linear) and MaxOut
architectures. It also suggests novel optimization challenges, while making the
link with $\ell_1$ minimization and sparsity-promoting techniques explicit.
