  Latent variable models have accumulated a considerable amount of interest
from the industry and academia for their versatility in a wide range of
applications. A large amount of effort has been made to develop systems that is
able to extend the systems to a large scale, in the hope to make use of them on
industry scale data. In this paper, we describe a system that operates at a
scale orders of magnitude higher than previous works, and an order of magnitude
faster than state-of-the-art system at the same scale, at the same time showing
more robustness and more accurate results.
  Our system uses a number of advances in distributed inference: high
performance in synchronization of sufficient statistics with relaxed
consistency model; fast sampling, using the Metropolis-Hastings-Walker method
to overcome dense generative models; statistical modeling, moving beyond Latent
Dirichlet Allocation (LDA) to Pitman-Yor distributions (PDP) and Hierarchical
Dirichlet Process (HDP) models; sophisticated parameter projection schemes, to
resolve the conflicts within the constraint between parameters arising from the
relaxed consistency model.
  This work significantly extends the domain of applicability of what is
commonly known as the Parameter Server. We obtain results with up to hundreds
billion oftokens, thousands of topics, and a vocabulary of a few million
token-types, using up to 60,000 processor cores operating on a production
cluster of a large Internet company. This demonstrates the feasibility to scale
to problems orders of magnitude larger than any previously published work.
