  We describe four algorithms for neural network training, each adapted to
different scalability constraints. These algorithms are mathematically
principled and invariant under a number of transformations in data and network
representation, from which performance is thus independent. These algorithms
are obtained from the setting of differential geometry, and are based on either
the natural gradient using the Fisher information matrix, or on Hessian
methods, scaled down in a specific way to allow for scalability while keeping
some of their key mathematical properties.
