  Self-supervised learning has significantly improved the performance of many
NLP tasks. However, how can self-supervised learning discover useful
representations, and why is it better than traditional approaches such as
probabilistic models are still largely unknown. In this paper, we focus on the
context of topic modeling and highlight a key advantage of self-supervised
learning - when applied to data generated by topic models, self-supervised
learning can be oblivious to the specific model, and hence is less susceptible
to model misspecification. In particular, we prove that commonly used
self-supervised objectives based on reconstruction or contrastive samples can
both recover useful posterior information for general topic models.
Empirically, we show that the same objectives can perform on par with posterior
inference using the correct model, while outperforming posterior inference
using misspecified models.
