  Recently, the relationship between automated and human evaluation of topic
models has been called into question. Method developers have staked the
efficacy of new topic model variants on automated measures, and their failure
to approximate human preferences places these models on uncertain ground.
Moreover, existing evaluation paradigms are often divorced from real-world use.
  Motivated by content analysis as a dominant real-world use case for topic
modeling, we analyze two related aspects of topic models that affect their
effectiveness and trustworthiness in practice for that purpose: the stability
of their estimates and the extent to which the model's discovered categories
align with human-determined categories in the data. We find that neural topic
models fare worse in both respects compared to an established classical method.
We take a step toward addressing both issues in tandem by demonstrating that a
straightforward ensembling method can reliably outperform the members of the
ensemble.
