  Second-order information is valuable for many applications but challenging to
compute. Several works focus on computing or approximating Hessian diagonals,
but even this simplification introduces significant additional costs compared
to computing a gradient. In the absence of efficient exact computation schemes
for Hessian diagonals, we revisit an early approximation scheme proposed by
Becker and LeCun (1989, BL89), which has a cost similar to gradients and
appears to have been overlooked by the community. We introduce HesScale, an
improvement over BL89, which adds negligible extra computation. On small
networks, we find that this improvement is of higher quality than all
alternatives, even those with theoretical guarantees, such as unbiasedness,
while being much cheaper to compute. We use this insight in reinforcement
learning problems where small networks are used and demonstrate HesScale in
second-order optimization and scaling the step-size parameter. In our
experiments, HesScale optimizes faster than existing methods and improves
stability through step-size scaling. These findings are promising for scaling
second-order methods in larger models in the future.
