  Deep learning models have shown their superior performance in various vision
tasks. However, the lack of precisely interpreting kernels in convolutional
neural networks (CNNs) is becoming one main obstacle to wide applications of
deep learning models in real scenarios. Although existing interpretation
methods may find certain visual patterns which are associated with the
activation of a specific kernel, those visual patterns may not be specific or
comprehensive enough for interpretation of a specific activation of kernel of
interest. In this paper, a simple yet effective optimization method is proposed
to interpret the activation of any kernel of interest in CNN models. The basic
idea is to simultaneously preserve the activation of the specific kernel and
suppress the activation of all other kernels at the same layer. In this way,
only visual information relevant to the activation of the specific kernel is
remained in the input. Consistent visual information from multiple modified
inputs would help users understand what kind of features are specifically
associated with specific kernel. Comprehensive evaluation shows that the
proposed method can help better interpret activation of specific kernels than
widely used methods, even when two kernels have very similar activation regions
from the same input image.
