  Clustering token-level contextualized word representations produces output
that shares many similarities with topic models for English text collections.
Unlike clusterings of vocabulary-level word embeddings, the resulting models
more naturally capture polysemy and can be used as a way of organizing
documents. We evaluate token clusterings trained from several different output
layers of popular contextualized language models. We find that BERT and GPT-2
produce high quality clusterings, but RoBERTa does not. These cluster models
are simple, reliable, and can perform as well as, if not better than, LDA topic
models, maintaining high topic quality even when the number of topics is large
relative to the size of the local collection.
