  Self-supervised learning (SSL) opens up huge opportunities for medical image
analysis that is well known for its lack of annotations. However, aggregating
massive (unlabeled) 3D medical images like computerized tomography (CT) remains
challenging due to its high imaging cost and privacy restrictions. In this
paper, we advocate bringing a wealth of 2D images like chest X-rays as
compensation for the lack of 3D data, aiming to build a universal medical
self-supervised representation learning framework, called UniMiSS. The
following problem is how to break the dimensionality barrier, \ie, making it
possible to perform SSL with both 2D and 3D images? To achieve this, we design
a pyramid U-like medical Transformer (MiT). It is composed of the switchable
patch embedding (SPE) module and Transformers. The SPE module adaptively
switches to either 2D or 3D patch embedding, depending on the input dimension.
The embedded patches are converted into a sequence regardless of their original
dimensions. The Transformers model the long-term dependencies in a
sequence-to-sequence manner, thus enabling UniMiSS to learn representations
from both 2D and 3D images. With the MiT as the backbone, we perform the
UniMiSS in a self-distillation manner. We conduct expensive experiments on six
3D/2D medical image analysis tasks, including segmentation and classification.
The results show that the proposed UniMiSS achieves promising performance on
various downstream tasks, outperforming the ImageNet pre-training and other
advanced SSL counterparts substantially. Code is available at
\def\UrlFont{\rm\small\ttfamily}
\url{https://github.com/YtongXie/UniMiSS-code}.
