  What neural networks learn depends fundamentally on the geometry of the
underlying loss function. We study how different regularizers affect the
geometry of this function. One of the most basic geometric properties of a
smooth function is whether it is Morse or not. For nonlinear deep neural
networks, the unregularized loss function $L$ is typically not Morse. We
consider several different regularizers, including weight decay, and study for
which regularizers the regularized function $L_\epsilon$ becomes Morse.
