  The success of Deep Learning methods is not well understood, though various
attempts at explaining it have been made, typically centered on properties of
stochastic gradient descent. Even less clear is why certain neural network
architectures perform better than others. We provide a potential opening with
the hypothesis that neural network training is a form of Genetic Programming.
