  Deep learning is gaining instant popularity in computer aided diagnosis of
COVID-19. Due to the high sensitivity of Computed Tomography (CT) to this
disease, CT-based COVID-19 detection with visual models is currently at the
forefront of medical imaging research. Outcomes published in this direction are
frequently claiming highly accurate detection under deep transfer learning.
This is leading medical technologists to believe that deep transfer learning is
the mainstream solution for the problem. However, our critical analysis of the
literature reveals an alarming performance disparity between different
published results. Hence, we conduct a systematic thorough investigation to
analyze the effectiveness of deep transfer learning for COVID-19 detection with
CT images. Exploring 14 state-of-the-art visual models with over 200 model
training sessions, we conclusively establish that the published literature is
frequently overestimating transfer learning performance for the problem, even
in the prestigious scientific sources. The roots of overestimation trace back
to inappropriate data curation. We also provide case studies that consider more
realistic scenarios, and establish transparent baselines for the problem. We
hope that our reproducible investigation will help in curbing hype-driven
claims for the critical problem of COVID-19 diagnosis, and pave the way for a
more transparent performance evaluation of techniques for CT-based COVID-19
detection.
