  The recent explosion in work on neural topic modeling has been criticized for
optimizing automated topic evaluation metrics at the expense of actual
meaningful topic identification. But human annotation remains expensive and
time-consuming. We propose LLM-based methods inspired by standard human topic
evaluations, in a family of metrics called Contextualized Topic Coherence
(CTC). We evaluate both a fully automated version as well as a semi-automated
CTC that allows human-centered evaluation of coherence while maintaining the
efficiency of automated methods. We evaluate CTC relative to five other metrics
on six topic models and find that it outperforms automated topic coherence
methods, works well on short documents, and is not susceptible to meaningless
but high-scoring topics.
