  Due to the high availability of large-scale annotated image datasets,
knowledge transfer from pre-trained models showed outstanding performance in
medical image classification. However, building a robust image classification
model for datasets with data irregularity or imbalanced classes can be a very
challenging task, especially in the medical imaging domain. In this paper, we
propose a novel deep convolutional neural network, we called Self Supervised
Super Sample Decomposition for Transfer learning (4S-DT) model. 4S-DT
encourages a coarse-to-fine transfer learning from large-scale image
recognition tasks to a specific chest X-ray image classification task using a
generic self-supervised sample decomposition approach. Our main contribution is
a novel self-supervised learning mechanism guided by a super sample
decomposition of unlabelled chest X-ray images. 4S-DT helps in improving the
robustness of knowledge transformation via a downstream learning strategy with
a class-decomposition layer to simplify the local structure of the data. 4S-DT
can deal with any irregularities in the image dataset by investigating its
class boundaries using a downstream class-decomposition mechanism. We used
50,000 unlabelled chest X-ray images to achieve our coarse-to-fine transfer
learning with an application to COVID-19 detection, as an exemplar. 4S-DT has
achieved a high accuracy of 99.8% (95% CI: 99.44%, 99.98%) in the detection of
COVID-19 cases on a large dataset and an accuracy of 97.54% (95%$ CI: 96.22%,
98.91%) on an extended test set enriched by augmented images of a small
dataset, out of which all real COVID-19 cases were detected, which was the
highest accuracy obtained when compared to other methods.
