  This study investigates the impact of self-supervised pretraining of 3D
semantic segmentation models on a large-scale, domain-specific dataset. We
introduce BRAINS-45K, a dataset of 44,756 brain MRI volumes from public
sources, the largest public dataset available, and revisit a number of design
choices for pretraining modern segmentation architectures by simplifying and
optimizing state-of-the-art methods, and combining them with a novel
augmentation strategy. The resulting AMAES framework is based on
masked-image-modeling and intensity-based augmentation reversal and balances
memory usage, runtime, and finetuning performance. Using the popular U-Net and
the recent MedNeXt architecture as backbones, we evaluate the effect of
pretraining on three challenging downstream tasks, covering single-sequence,
low-resource settings, and out-of-domain generalization. The results highlight
that pretraining on the proposed dataset with AMAES significantly improves
segmentation performance in the majority of evaluated cases, and that it is
beneficial to pretrain the model with augmentations, despite pretraing on a
large-scale dataset. Code and model checkpoints for reproducing results, as
well as the BRAINS-45K dataset are available at
\url{https://github.com/asbjrnmunk/amaes}.
