  Supervised topic models simultaneously model the latent topic structure of
large collections of documents and a response variable associated with each
document. Existing inference methods are based on variational approximation or
Monte Carlo sampling, which often suffers from the local minimum defect.
Spectral methods have been applied to learn unsupervised topic models, such as
latent Dirichlet allocation (LDA), with provable guarantees. This paper
investigates the possibility of applying spectral methods to recover the
parameters of supervised LDA (sLDA). We first present a two-stage spectral
method, which recovers the parameters of LDA followed by a power update method
to recover the regression model parameters. Then, we further present a
single-phase spectral algorithm to jointly recover the topic distribution
matrix as well as the regression weights. Our spectral algorithms are provably
correct and computationally efficient. We prove a sample complexity bound for
each algorithm and subsequently derive a sufficient condition for the
identifiability of sLDA. Thorough experiments on synthetic and real-world
datasets verify the theory and demonstrate the practical effectiveness of the
spectral algorithms. In fact, our results on a large-scale review rating
dataset demonstrate that our single-phase spectral algorithm alone gets
comparable or even better performance than state-of-the-art methods, while
previous work on spectral methods has rarely reported such promising
performance.
