  Recent numerical experiments have demonstrated that the choice of
optimization geometry used during training can impact generalization
performance when learning expressive nonlinear model classes such as deep
neural networks. These observations have important implications for modern deep
learning but remain poorly understood due to the difficulty of the associated
nonconvex optimization problem. Towards an understanding of this phenomenon, we
analyze a family of pseudogradient methods for learning generalized linear
models under the square loss - a simplified problem containing both
nonlinearity in the model parameters and nonconvexity of the optimization which
admits a single neuron as a special case. We prove non-asymptotic bounds on the
generalization error that sharply characterize how the interplay between the
optimization geometry and the feature space geometry sets the out-of-sample
performance of the learned model. Experimentally, selecting the optimization
geometry as suggested by our theory leads to improved performance in
generalized linear model estimation problems such as nonlinear and nonconvex
variants of sparse vector recovery and low-rank matrix sensing.
