  Pretraining vision transformers (ViT) with attention guided masked image
modeling (MIM) has shown to increase downstream accuracy for natural image
analysis. Hierarchical shifted window (Swin) transformer, often used in medical
image analysis cannot use attention guided masking as it lacks an explicit
[CLS] token, needed for computing attention maps for selective masking. We thus
enhanced Swin with semantic class attention. We developed a co-distilled Swin
transformer that combines a noisy momentum updated teacher to guide selective
masking for MIM. Our approach called \textsc{s}e\textsc{m}antic
\textsc{a}ttention guided co-distillation with noisy teacher
\textsc{r}egularized Swin \textsc{T}rans\textsc{F}ormer (SMARTFormer) was
applied for analyzing 3D computed tomography datasets with lung nodules and
malignant lung cancers (LC). We also analyzed the impact of semantic attention
and noisy teacher on pretraining and downstream accuracy. SMARTFormer
classified lesions (malignant from benign) with a high accuracy of 0.895 of
1000 nodules, predicted LC treatment response with accuracy of 0.74, and
achieved high accuracies even in limited data regimes. Pretraining with
semantic attention and noisy teacher improved ability to distinguish
semantically meaningful structures such as organs in a unsupervised clustering
task and localize abnormal structures like tumors. Code, models will be made
available through GitHub upon paper acceptance.
