  While natural gradients have been widely studied from both theoretical and
empirical perspectives, we argue that some fundamental theoretical issues
regarding the existence of gradients in infinite dimensional function spaces
remain underexplored. We address these issues by providing a geometric
perspective and mathematical framework for studying natural gradient that is
more complete and rigorous than existing studies. Our results also establish
new connections between natural gradients and RKHS theory, and specifically to
the Neural Tangent Kernel (NTK). Based on our theoretical framework, we derive
a new family of natural gradients induced by Sobolev metrics and develop
computational techniques for efficient approximation in practice. Preliminary
experimental results reveal the potential of this new natural gradient variant.
