  There are not many large medical image datasets available. For these
datasets, too small deep learning models can't learn useful features, so they
don't work well due to underfitting, and too big models tend to overfit the
limited data. As a result, there is a compromise between the two issues. This
paper proposes a training strategy Medi-CAT to overcome the underfitting and
overfitting phenomena in medical imaging datasets. Specifically, the proposed
training methodology employs large pre-trained vision transformers to overcome
underfitting and adversarial and contrastive learning techniques to prevent
overfitting. The proposed method is trained and evaluated on four medical image
classification datasets from the MedMNIST collection. Our experimental results
indicate that the proposed approach improves the accuracy up to 2% on three
benchmark datasets compared to well-known approaches, whereas it increases the
performance up to 4.1% over the baseline methods.
