  Deep Neural Networks (DNNs) have been shown to be susceptible to Trojan
attacks. Neural Trojan is a type of targeted poisoning attack that embeds the
backdoor into the victim and is activated by the trigger in the input space.
The increasing deployment of DNNs in critical systems and the surge of
outsourcing DNN training (which makes Trojan attack easier) makes the detection
of Trojan attacks necessary. While Neural Trojan detection has been studied in
the image domain, there is a lack of solutions in the NLP domain. In this
paper, we propose a model-level Trojan detection framework by analyzing the
deviation of the model output when we introduce a specially crafted
perturbation to the input. Particularly, we extract the model's responses to
perturbed inputs as the `signature' of the model and train a meta-classifier to
determine if a model is Trojaned based on its signature. We demonstrate the
effectiveness of our proposed method on both a dataset of NLP models we create
and a public dataset of Trojaned NLP models from TrojAI. Furthermore, we
propose a lightweight variant of our detection method that reduces the
detection time while preserving the detection rates.
