  Stochastic models with global parameters and latent variables are common, and
for which variational inference (VI) is popular. However, existing methods are
often either slow or inaccurate in high dimensions. We suggest a fast and
accurate VI method for this case that employs a well-defined natural gradient
variational optimization that targets the joint posterior of the global
parameters and latent variables. It is a hybrid method, where at each step the
global parameters are updated using the natural gradient and the latent
variables are generated from their conditional posterior. A fast to compute
expression for the Tikhonov damped Fisher information matrix is used, along
with the re-parameterization trick, to provide a stable natural gradient. We
apply the approach to deep mixed models, which are an emerging class of
Bayesian neural networks with random output layer coefficients to allow for
heterogeneity. A range of simulations show that using the natural gradient is
substantially more efficient than using the ordinary gradient, and that the
approach is faster and more accurate than two cutting-edge natural gradient VI
methods. In a financial application we show that accounting for industry level
heterogeneity using the deep mixed model improves the accuracy of asset pricing
models. MATLAB code to implement the method can be found at:
https://github.com/WeibenZhang07/NG-HVI.
