  Recent advances in self-supervised learning with instance-level contrastive
objectives facilitate unsupervised clustering. However, a standalone datum is
not perceiving the context of the holistic cluster, and may undergo sub-optimal
assignment. In this paper, we extend the mainstream contrastive learning
paradigm to a cluster-level scheme, where all the data subjected to the same
cluster contribute to a unified representation that encodes the context of each
data group. Contrastive learning with this representation then rewards the
assignment of each datum. To implement this vision, we propose twin-contrast
clustering (TCC). We define a set of categorical variables as clustering
assignment confidence, which links the instance-level learning track with the
cluster-level one. On one hand, with the corresponding assignment variables
being the weight, a weighted aggregation along the data points implements the
set representation of a cluster. We further propose heuristic cluster
augmentation equivalents to enable cluster-level contrastive learning. On the
other hand, we derive the evidence lower-bound of the instance-level
contrastive objective with the assignments. By reparametrizing the assignment
variables, TCC is trained end-to-end, requiring no alternating steps. Extensive
experiments show that TCC outperforms the state-of-the-art on challenging
benchmarks.
