  Second-order methods have shown state-of-the-art performance for optimizing
deep neural networks. Nonetheless, their large memory requirement and high
computational complexity, compared to first-order methods, hinder their
versatility in a typical low-budget setup. This paper introduces a general
framework of layerwise loss construction for multilayer neural networks that
achieves a performance closer to second-order methods while utilizing
first-order optimizers only. Our methodology lies upon a three-component loss,
target, and regularizer combination, for which altering each component results
in a new update rule. We provide examples using squared loss and layerwise
Bregman divergences induced by the convex integral functions of various
transfer functions. Our experiments on benchmark models and datasets validate
the efficacy of our new approach, reducing the gap between first-order and
second-order optimizers.
