  While deep learning techniques have provided the state-of-the-art performance
in various clinical tasks, explainability regarding their decision-making
process can greatly enhance the credence of these methods for safer and quicker
clinical adoption. With high flexibility, Gradient-weighted Class Activation
Mapping (Grad-CAM) has been widely adopted to offer intuitive visual
interpretation of various deep learning models' reasoning processes in
computer-assisted diagnosis. However, despite the popularity of the technique,
there is still a lack of systematic study on Grad-CAM's performance on
different deep learning architectures. In this study, we investigate its
robustness and effectiveness across different popular deep learning models,
with a focus on the impact of the networks' depths and architecture types, by
using a case study of automatic pneumothorax diagnosis in X-ray scans. Our
results show that deeper neural networks do not necessarily contribute to a
strong improvement of pneumothorax diagnosis accuracy, and the effectiveness of
GradCAM also varies among different network architectures.
