  We propose a novel generative model to explore both local and global context
for joint learning topics and topic-specific word embeddings. In particular, we
assume that global latent topics are shared across documents, a word is
generated by a hidden semantic vector encoding its contextual semantic meaning,
and its context words are generated conditional on both the hidden semantic
vector and global latent topics. Topics are trained jointly with the word
embeddings. The trained model maps words to topic-dependent embeddings, which
naturally addresses the issue of word polysemy. Experimental results show that
the proposed model outperforms the word-level embedding methods in both word
similarity evaluation and word sense disambiguation. Furthermore, the model
also extracts more coherent topics compared with existing neural topic models
or other models for joint learning of topics and word embeddings. Finally, the
model can be easily integrated with existing deep contextualized word embedding
learning methods to further improve the performance of downstream tasks such as
sentiment classification.
