  Recent advancements in foundation models, typically trained with
self-supervised learning on large-scale and diverse datasets, have shown great
potential in medical image analysis. However, due to the significant spatial
heterogeneity of medical imaging data, current models must tailor specific
structures for different datasets, making it challenging to leverage the
abundant unlabeled data. In this work, we propose a universal foundation model
for medical image analysis that processes images with heterogeneous spatial
properties using a unified structure. To accomplish this, we propose spatially
adaptive networks (SPAD-Nets), a family of networks that dynamically adjust the
structures to adapt to the spatial properties of input images, to build such a
universal foundation model. We pre-train a spatial adaptive visual tokenizer
(SPAD-VT) and then a spatial adaptive Vision Transformer (SPAD-ViT) via masked
image modeling (MIM) on 55 public medical image datasets. The pre-training data
comprises over 9 million image slices, representing the largest, most
comprehensive, and most diverse dataset to our knowledge for pre-training
universal foundation models for medical image analysis. The experimental
results on downstream medical image classification and segmentation tasks
demonstrate the superior performance and label efficiency of our model. Our
code is available at https://github.com/function2-llx/PUMIT.
