  Pretraining on large natural image classification datasets such as ImageNet
has aided model development on data-scarce 2D medical tasks. 3D medical tasks
often have much less data than 2D medical tasks, prompting practitioners to
rely on pretrained 2D models to featurize slices. However, these 2D models have
been surpassed by 3D models on 3D computer vision benchmarks since they do not
natively leverage cross-sectional or temporal information. In this study, we
explore whether natural video pretraining for 3D models can enable higher
performance on smaller datasets for 3D medical tasks. We demonstrate video
pretraining improves the average performance of seven 3D models on two chest CT
datasets, regardless of finetuning dataset size, and that video pretraining
allows 3D models to outperform 2D baselines. Lastly, we observe that
pretraining on the large-scale out-of-domain Kinetics dataset improves
performance more than pretraining on a typically-sized in-domain CT dataset.
Our results show consistent benefits of video pretraining across a wide array
of architectures, tasks, and training dataset sizes, supporting a shift from
small-scale in-domain pretraining to large-scale out-of-domain pretraining for
3D medical tasks. Our code is available at:
https://github.com/rajpurkarlab/chest-ct-pretraining
