  Despite the remarkable success of Large Language Models (LLMs) in text
understanding and generation, their potential for text clustering tasks remains
underexplored. We observed that powerful closed-source LLMs provide good
quality clusterings of entity sets but are not scalable due to the massive
compute power required and the associated costs. Thus, we propose CACTUS
(Context-Aware ClusTering with aUgmented triplet losS), a systematic approach
that leverages open-source LLMs for efficient and effective supervised
clustering of entity subsets, particularly focusing on text-based entities.
Existing text clustering methods fail to effectively capture the context
provided by the entity subset. Moreover, though there are several language
modeling based approaches for clustering, very few are designed for the task of
supervised clustering. This paper introduces a novel approach towards
clustering entity subsets using LLMs by capturing context via a scalable
inter-entity attention mechanism. We propose a novel augmented triplet loss
function tailored for supervised clustering, which addresses the inherent
challenges of directly applying the triplet loss to this problem. Furthermore,
we introduce a self-supervised clustering task based on text augmentation
techniques to improve the generalization of our model. For evaluation, we
collect ground truth clusterings from a closed-source LLM and transfer this
knowledge to an open-source LLM under the supervised clustering framework,
allowing a faster and cheaper open-source model to perform the same task.
Experiments on various e-commerce query and product clustering datasets
demonstrate that our proposed approach significantly outperforms existing
unsupervised and supervised baselines under various external clustering
evaluation metrics.
