  Supervised topic models with a logistic likelihood have two issues that
potentially limit their practical use: 1) response variables are usually
over-weighted by document word counts; and 2) existing variational inference
methods make strict mean-field assumptions. We address these issues by: 1)
introducing a regularization constant to better balance the two parts based on
an optimization formulation of Bayesian inference; and 2) developing a simple
Gibbs sampling algorithm by introducing auxiliary Polya-Gamma variables and
collapsing out Dirichlet variables. Our augment-and-collapse sampling algorithm
has analytical forms of each conditional distribution without making any
restricting assumptions and can be easily parallelized. Empirical results
demonstrate significant improvements on prediction performance and time
efficiency.
