  Backdoor attacks (BAs) are an emerging form of adversarial attack typically
against deep neural network image classifiers. The attacker aims to have the
classifier learn to classify to a target class when test images from one or
more source classes contain a backdoor pattern, while maintaining high accuracy
on all clean test images. Reverse-Engineering-based Defenses (REDs) against BAs
do not require access to the training set but only to an independent clean
dataset. Unfortunately, most existing REDs rely on an unrealistic assumption
that all classes except the target class are source classes of the attack. REDs
that do not rely on this assumption often require a large set of clean images
and heavy computation. In this paper, we propose a Lagrangian-based RED (L-RED)
that does not require knowledge of the number of source classes (or whether an
attack is present). Our defense requires very few clean images to effectively
detect BAs and is computationally efficient. Notably, we detect 56 out of 60
BAs using only two clean images per class in our experiments on CIFAR-10.
