  Optimization problems that include regularization functions in their
objectives are regularly solved in many applications. When one seeks
second-order methods for such problems, it may be desirable to exploit specific
properties of some of these regularization functions when accounting for
curvature information in the solution steps to speed up convergence. In this
paper, we propose the SCORE (self-concordant regularization) framework for
unconstrained minimization problems which incorporates second-order information
in the Newton-decrement framework for convex optimization. We propose the
generalized Gauss-Newton with Self-Concordant Regularization (GGN-SCORE)
algorithm that updates the minimization variables each time it receives a new
input batch. The proposed algorithm exploits the structure of the second-order
information in the Hessian matrix, thereby reducing computational overhead.
GGN-SCORE demonstrates how to speed up convergence while also improving model
generalization for problems that involve regularized minimization under the
proposed SCORE framework. Numerical experiments show the efficiency of our
method and its fast convergence, which compare favorably against baseline
first-order and quasi-Newton methods. Additional experiments involving
non-convex (overparameterized) neural network training problems show that the
proposed method is promising for non-convex optimization.
