  This paper introduces a method for efficiently approximating the inverse of
the Fisher information matrix, a crucial step in achieving effective
variational Bayes inference. A notable aspect of our approach is the avoidance
of analytically computing the Fisher information matrix and its explicit
inversion. Instead, we introduce an iterative procedure for generating a
sequence of matrices that converge to the inverse of Fisher information. The
natural gradient variational Bayes algorithm without analytic expression of the
Fisher matrix and its inversion is provably convergent and achieves a
convergence rate of order O(log s/s), with s the number of iterations. We also
obtain a central limit theorem for the iterates. Implementation of our method
does not require storage of large matrices, and achieves a linear complexity in
the number of variational parameters. Our algorithm exhibits versatility,
making it applicable across a diverse array of variational Bayes domains,
including Gaussian approximation and normalizing flow Variational Bayes. We
offer a range of numerical examples to demonstrate the efficiency and
reliability of the proposed variational Bayes method.
