  Distillation-based self-supervised learning typically leads to more
compressed representations due to its radical clustering process and the
implementation of a sharper target distribution. To overcome this limitation
and preserve more information from input, we introduce UDI, conceptualized as
Unsqueezed Distillation-based self-supervised learning (SSL). UDI enriches the
learned representation by encouraging multimodal prediction distilled from a
consolidated profile of local predictions that are derived via stratified
sampling. Our evaluations show that UDI not only promotes semantically
meaningful representations at instance level, delivering superior or
competitive results to state-of-the-art SSL methods in image classification,
but also effectively preserves the nuisance of input, which yields significant
improvement in dense prediction tasks, including object detection and
segmentation. Additionally, UDI performs competitively in low-shot image
classification, improving the scalability of joint-embedding pipelines. Various
visualizations and ablation studies are presented to further elucidate the
mechanisms behind UDI. Our source code is available at
https://github.com/ISL-CV/udi.
