  Convolutional Neural Networks (CNNs) have reigned for a decade as the de
facto approach to automated medical image diagnosis, pushing the
state-of-the-art in classification, detection and segmentation tasks. Over the
last years, vision transformers (ViTs) have appeared as a competitive
alternative to CNNs, yielding impressive levels of performance in the natural
image domain, while possessing several interesting properties that could prove
beneficial for medical imaging tasks. In this work, we explore the benefits and
drawbacks of transformer-based models for medical image classification. We
conduct a series of experiments on several standard 2D medical image benchmark
datasets and tasks. Our findings show that, while CNNs perform better if
trained from scratch, off-the-shelf vision transformers can perform on par with
CNNs when pretrained on ImageNet, both in a supervised and self-supervised
setting, rendering them as a viable alternative to CNNs.
