  We investigate the function-space optimality (specifically, the Banach-space
optimality) of a large class of shallow neural architectures with multivariate
nonlinearities/activation functions. To that end, we construct a new family of
Banach spaces defined via a regularization operator, the $k$-plane transform,
and a sparsity-promoting norm. We prove a representer theorem that states that
the solution sets to learning problems posed over these Banach spaces are
completely characterized by neural architectures with multivariate
nonlinearities. These optimal architectures have skip connections and are
tightly connected to orthogonal weight normalization and multi-index models,
both of which have received recent interest in the neural network community.
Our framework is compatible with a number of classical nonlinearities including
the rectified linear unit (ReLU) activation function, the norm activation
function, and the radial basis functions found in the theory of
thin-plate/polyharmonic splines. We also show that the underlying spaces are
special instances of reproducing kernel Banach spaces and variation spaces. Our
results shed light on the regularity of functions learned by neural networks
trained on data, particularly with multivariate nonlinearities, and provide new
theoretical motivation for several architectural choices found in practice.
