  Effective representation learning is critical for short text clustering due
to the sparse, high-dimensional and noise attributes of short text corpus.
Existing pre-trained models (e.g., Word2vec and BERT) have greatly improved the
expressiveness for short text representations with more condensed,
low-dimensional and continuous features compared to the traditional
Bag-of-Words (BoW) model. However, these models are trained for general
purposes and thus are suboptimal for the short text clustering task. In this
paper, we propose two methods to exploit the unsupervised autoencoder (AE)
framework to further tune the short text representations based on these
pre-trained text models for optimal clustering performance. In our first method
Structural Text Network Graph Autoencoder (STN-GAE), we exploit the structural
text information among the corpus by constructing a text network, and then
adopt graph convolutional network as encoder to fuse the structural features
with the pre-trained text features for text representation learning. In our
second method Soft Cluster Assignment Autoencoder (SCA-AE), we adopt an extra
soft cluster assignment constraint on the latent space of autoencoder to
encourage the learned text representations to be more clustering-friendly. We
tested two methods on seven popular short text datasets, and the experimental
results show that when only using the pre-trained model for short text
clustering, BERT performs better than BoW and Word2vec. However, as long as we
further tune the pre-trained representations, the proposed method like SCA-AE
can greatly increase the clustering performance, and the accuracy improvement
compared to use BERT alone could reach as much as 14\%.
