  Deep neural networks (DNNs) are increasingly deployed in different
applications to achieve state-of-the-art performance. However, they are often
applied as a black box with limited understanding of what knowledge the model
has learned from the data. In this paper, we focus on image classification and
propose a method to visualize and understand the class-wise knowledge
(patterns) learned by DNNs under three different settings including natural,
backdoor and adversarial. Different to existing visualization methods, our
method searches for a single predictive pattern in the pixel space to represent
the knowledge learned by the model for each class. Based on the proposed
method, we show that DNNs trained on natural (clean) data learn abstract shapes
along with some texture, and backdoored models learn a suspicious pattern for
the backdoored class. Interestingly, the phenomenon that DNNs can learn a
single predictive pattern for each class indicates that DNNs can learn a
backdoor even from clean data, and the pattern itself is a backdoor trigger. In
the adversarial setting, we show that adversarially trained models tend to
learn more simplified shape patterns. Our method can serve as a useful tool to
better understand the knowledge learned by DNNs on different datasets under
different settings.
