  As an emerging approach to explore the vulnerability of deep neural networks
(DNNs), backdoor learning has attracted increasing interest in recent years,
and many seminal backdoor attack and defense algorithms are being developed
successively or concurrently, in the status of a rapid arms race. However,
mainly due to the diverse settings, and the difficulties of implementation and
reproducibility of existing works, there is a lack of a unified and
standardized benchmark of backdoor learning, causing unfair comparisons or
unreliable conclusions (e.g., misleading, biased or even false conclusions).
Consequently, it is difficult to evaluate the current progress and design the
future development roadmap of this literature. To alleviate this dilemma, we
build a comprehensive benchmark of backdoor learning called BackdoorBench. Our
benchmark makes three valuable contributions to the research community. 1) We
provide an integrated implementation of state-of-the-art (SOTA) backdoor
learning algorithms (currently including 20 attack and 32 defense algorithms),
based on an extensible modular-based codebase. 2) We conduct comprehensive
evaluations with 5 poisoning ratios, based on 4 models and 4 datasets, leading
to 11,492 pairs of attack-against-defense evaluations in total. 3) Based on
above evaluations, we present abundant analysis from 10 perspectives via 18
useful analysis tools, and provide several inspiring insights about backdoor
learning. We hope that our efforts could build a solid foundation of backdoor
learning to facilitate researchers to investigate existing algorithms, develop
more innovative algorithms, and explore the intrinsic mechanism of backdoor
learning. Finally, we have created a user-friendly website at
http://backdoorbench.com, which collects all important information of
BackdoorBench, including codebase, docs, leaderboard, and model Zoo.
