  There is a longstanding debate whether the Kolmogorov-Arnold representation
theorem can explain the use of more than one hidden layer in neural networks.
The Kolmogorov-Arnold representation decomposes a multivariate function into an
interior and an outer function and therefore has indeed a similar structure as
a neural network with two hidden layers. But there are distinctive differences.
One of the main obstacles is that the outer function depends on the represented
function and can be wildly varying even if the represented function is smooth.
We derive modifications of the Kolmogorov-Arnold representation that transfer
smoothness properties of the represented function to the outer function and can
be well approximated by ReLU networks. It appears that instead of two hidden
layers, a more natural interpretation of the Kolmogorov-Arnold representation
is that of a deep neural network where most of the layers are required to
approximate the interior function.
