  In this paper, we propose new structured second-order methods and structured
adaptive-gradient methods obtained by performing natural-gradient descent on
structured parameter spaces. Natural-gradient descent is an attractive approach
to design new algorithms in many settings such as gradient-free,
adaptive-gradient, and second-order methods. Our structured methods not only
enjoy a structural invariance but also admit a simple expression. Finally, we
test the efficiency of our proposed methods on both deterministic non-convex
problems and deep learning problems.
