  Large language models (LLM)'s are increasingly used for topic modeling
outperforming classical topic models such as LDA. Commonly, pre-trained LLM
encoders such as BERT are used out-of-the-box despite the fact that fine-tuning
is known to improve LLMs considerably. The challenge lies in obtaining a
suitable (labeled) dataset for fine-tuning. In this paper, we use the recent
idea to use bag of sentences as the elementary unit in computing topics. In
turn, we derive an approach FT-Topic to perform unsupervised fine-tuning
relying primarily on two steps for constructing a training dataset in an
automatic fashion. First, a heuristic method to identifies pairs of sentence
groups that are either assumed to be of the same or different topics. Second,
we remove sentence pairs that are likely labeled incorrectly. The dataset is
then used to fine-tune an encoder LLM, which can be leveraged by any topic
modeling approach using embeddings. However, in this work, we demonstrate its
effectiveness by deriving a novel state-of-the-art topic modeling method called
SenClu, which achieves fast inference through an expectation-maximization
algorithm and hard assignments of sentence groups to a single topic, while
giving users the possibility to encode prior knowledge on the topic-document
distribution. Code is at \url{https://github.com/JohnTailor/FT-Topic}
