  A high degree of topical diversity is often considered to be an important
characteristic of interesting text documents. A recent proposal for measuring
topical diversity identifies three distributions for assessing the diversity of
documents: distributions of words within documents, words within topics, and
topics within documents. Topic models play a central role in this approach and,
hence, their quality is crucial to the efficacy of measuring topical diversity.
The quality of topic models is affected by two causes: generality and impurity
of topics. General topics only include common information of a background
corpus and are assigned to most of the documents. Impure topics contain words
that are not related to the topic. Impurity lowers the interpretability of
topic models. Impure topics are likely to get assigned to documents
erroneously. We propose a hierarchical re-estimation process aimed at removing
generality and impurity. Our approach has three re-estimation components: (1)
document re-estimation, which removes general words from the documents; (2)
topic re-estimation, which re-estimates the distribution over words of each
topic; and (3) topic assignment re-estimation, which re-estimates for each
document its distributions over topics. For measuring topical diversity of text
documents, our HiTR approach improves over the state-of-the-art measured on
PubMed dataset.
