  It has been observed that design choices of neural networks are often crucial
for their successful optimization. In this article, we therefore discuss the
question if it is always possible to redesign a neural network so that it
trains well with gradient descent. This yields the following universality
result: If, for a given network, there is any algorithm that can find good
network weights for a classification task, then there exists an extension of
this network that reproduces these weights and the corresponding forward output
by mere gradient descent training. The construction is not intended for
practical computations, but it provides some orientation on the possibilities
of meta-learning and related approaches.
