  Deep neural networks have achieved state-of-the-art performance on various
tasks. However, lack of interpretability and transparency makes it easier for
malicious attackers to inject trojan backdoor into the neural networks, which
will make the model behave abnormally when a backdoor sample with a specific
trigger is input. In this paper, we propose NeuronInspect, a framework to
detect trojan backdoors in deep neural networks via output explanation
techniques. NeuronInspect first identifies the existence of backdoor attack
targets by generating the explanation heatmap of the output layer. We observe
that generated heatmaps from clean and backdoored models have different
characteristics. Therefore we extract features that measure the attributes of
explanations from an attacked model namely: sparse, smooth and persistent. We
combine these features and use outlier detection to figure out the outliers,
which is the set of attack targets. We demonstrate the effectiveness and
efficiency of NeuronInspect on MNIST digit recognition dataset and GTSRB
traffic sign recognition dataset. We extensively evaluate NeuronInspect on
different attack scenarios and prove better robustness and effectiveness over
state-of-the-art trojan backdoor detection techniques Neural Cleanse by a great
margin.
