  Deep neural networks (DNNs) are proved to be vulnerable against backdoor
attacks. A backdoor is often embedded in the target DNNs through injecting a
backdoor trigger into training examples, which can cause the target DNNs
misclassify an input attached with the backdoor trigger. Existing backdoor
detection methods often require the access to the original poisoned training
data, the parameters of the target DNNs, or the predictive confidence for each
given input, which are impractical in many real-world applications, e.g.,
on-device deployed DNNs. We address the black-box hard-label backdoor detection
problem where the DNN is fully black-box and only its final output label is
accessible. We approach this problem from the optimization perspective and show
that the objective of backdoor detection is bounded by an adversarial
objective. Further theoretical and empirical studies reveal that this
adversarial objective leads to a solution with highly skewed distribution; a
singularity is often observed in the adversarial map of a backdoor-infected
example, which we call the adversarial singularity phenomenon. Based on this
observation, we propose the adversarial extreme value analysis(AEVA) to detect
backdoors in black-box neural networks. AEVA is based on an extreme value
analysis of the adversarial map, computed from the monte-carlo gradient
estimation. Evidenced by extensive experiments across multiple popular tasks
and backdoor attacks, our approach is shown effective in detecting backdoor
attacks under the black-box hard-label scenarios.
