  A matrix free and a low rank approximation preconditioner are proposed to
accelerate the convergence of stochastic gradient descent (SGD) by exploiting
curvature information sampled from Hessian-vector products or finite
differences of parameters and gradients similar to the BFGS algorithm. Both
preconditioners are fitted with an online updating manner minimizing a
criterion that is free of line search and robust to stochastic gradient noise,
and further constrained to be on certain connected Lie groups to preserve their
corresponding symmetry or invariance, e.g., orientation of coordinates by the
connected general linear group with positive determinants. The Lie group's
equivariance property facilitates preconditioner fitting, and its invariance
property saves any need of damping, which is common in second-order optimizers,
but difficult to tune. The learning rate for parameter updating and step size
for preconditioner fitting are naturally normalized, and their default values
work well in most situations.
