  We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that
conditions the gradient using selected second-order information and has an
asymptotically vanishing computational overhead, assuming a batch size smaller
than the number of neurons. We show that it is possible to compute a good
conditioner based on only the input to a respective layer without a substantial
computational overhead. The proposed method allows effective training even in
small-batch stochastic regimes, which makes it competitive to first-order as
well as second-order methods.
