  Deep neural networks are being widely deployed for many critical tasks due to
their high classification accuracy. In many cases, pre-trained models are
sourced from vendors who may have disrupted the training pipeline to insert
Trojan behaviors into the models. These malicious behaviors can be triggered at
the adversary's will and hence, cause a serious threat to the widespread
deployment of deep models. We propose a method to verify if a pre-trained model
is Trojaned or benign. Our method captures fingerprints of neural networks in
the form of adversarial perturbations learned from the network gradients.
Inserting backdoors into a network alters its decision boundaries which are
effectively encoded in their adversarial perturbations. We train a two stream
network for Trojan detection from its global ($L_\infty$ and $L_2$ bounded)
perturbations and the localized region of high energy within each perturbation.
The former encodes decision boundaries of the network and latter encodes the
unknown trigger shape. We also propose an anomaly detection method to identify
the target class in a Trojaned network. Our methods are invariant to the
trigger type, trigger size, training data and network architecture. We evaluate
our methods on MNIST, NIST-Round0 and NIST-Round1 datasets, with up to 1,000
pre-trained models making this the largest study to date on Trojaned network
detection, and achieve over 92\% detection accuracy to set the new
state-of-the-art.
