  We introduce Contrastive Gaussian Clustering, a novel approach capable of
provide segmentation masks from any viewpoint and of enabling 3D segmentation
of the scene. Recent works in novel-view synthesis have shown how to model the
appearance of a scene via a cloud of 3D Gaussians, and how to generate accurate
images from a given viewpoint by projecting on it the Gaussians before $\alpha$
blending their color. Following this example, we train a model to include also
a segmentation feature vector for each Gaussian. These can then be used for 3D
scene segmentation, by clustering Gaussians according to their feature vectors;
and to generate 2D segmentation masks, by projecting the Gaussians on a plane
and $\alpha$ blending over their segmentation features. Using a combination of
contrastive learning and spatial regularization, our method can be trained on
inconsistent 2D segmentation masks, and still learn to generate segmentation
masks consistent across all views. Moreover, the resulting model is extremely
accurate, improving the IoU accuracy of the predicted masks by $+8\%$ over the
state of the art. Code and trained models will be released soon.
