  The generalized Gauss-Newton (GGN) optimization method incorporates curvature
estimates into its solution steps, and provides a good approximation to the
Newton method for large-scale optimization problems. GGN has been found
particularly interesting for practical training of deep neural networks, not
only for its impressive convergence speed, but also for its close relation with
neural tangent kernel regression, which is central to recent studies that aim
to understand the optimization and generalization properties of neural
networks. This work studies a GGN method for optimizing a two-layer neural
network with explicit regularization. In particular, we consider a class of
generalized self-concordant (GSC) functions that provide smooth approximations
to commonly-used penalty terms in the objective function of the optimization
problem. This approach provides an adaptive learning rate selection technique
that requires little to no tuning for optimal performance. We study the
convergence of the two-layer neural network, considered to be
overparameterized, in the optimization loop of the resulting GGN method for a
given scaling of the network parameters. Our numerical experiments highlight
specific aspects of GSC regularization that help to improve generalization of
the optimized neural network. The code to reproduce the experimental results is
available at https://github.com/adeyemiadeoye/ggn-score-nn.
