  The convolutional neural network (CNN) has become a powerful tool for various
biomedical image analysis tasks, but there is a lack of visual explanation for
the machinery of CNNs. In this paper, we present a novel algorithm,
Respond-weighted Class Activation Mapping (Respond-CAM), for making CNN-based
models interpretable by visualizing input regions that are important for
predictions, especially for biomedical 3D imaging data inputs. Our method uses
the gradients of any target concept (e.g. the score of target class) that flows
into a convolutional layer. The weighted feature maps are combined to produce a
heatmap that highlights the important regions in the image for predicting the
target concept. We prove a preferable sum-to-score property of the Respond-CAM
and verify its significant improvement on 3D images from the current
state-of-the-art approach. Our tests on Cellular Electron Cryo-Tomography 3D
images show that Respond-CAM achieves superior performance on visualizing the
CNNs with 3D biomedical images inputs, and is able to get reasonably good
results on visualizing the CNNs with natural image inputs. The Respond-CAM is
an efficient and reliable approach for visualizing the CNN machinery, and is
applicable to a wide variety of CNN model families and image analysis tasks.
