  Stochastic variational inference (SVI) employs stochastic optimization to
scale up Bayesian computation to massive data. Since SVI is at its core a
stochastic gradient-based algorithm, horizontal parallelism can be harnessed to
allow larger scale inference. We propose a lock-free parallel implementation
for SVI which allows distributed computations over multiple slaves in an
asynchronous style. We show that our implementation leads to linear speed-up
while guaranteeing an asymptotic ergodic convergence rate $O(1/\sqrt(T)$ )
given that the number of slaves is bounded by $\sqrt(T)$ ($T$ is the total
number of iterations). The implementation is done in a high-performance
computing (HPC) environment using message passing interface (MPI) for python
(MPI4py). The extensive empirical evaluation shows that our parallel SVI is
lossless, performing comparably well to its counterpart serial SVI with linear
speed-up.
