  Neural ranking models (NRMs) have achieved promising results in information
retrieval. NRMs have also been shown to be vulnerable to adversarial examples.
A typical Word Substitution Ranking Attack (WSRA) against NRMs was proposed
recently, in which an attacker promotes a target document in rankings by adding
human-imperceptible perturbations to its text. This raises concerns when
deploying NRMs in real-world applications. Therefore, it is important to
develop techniques that defend against such attacks for NRMs. In empirical
defenses adversarial examples are found during training and used to augment the
training set. However, such methods offer no theoretical guarantee on the
models' robustness and may eventually be broken by other sophisticated WSRAs.
To escape this arms race, rigorous and provable certified defense methods for
NRMs are needed.
  To this end, we first define the \textit{Certified Top-$K$ Robustness} for
ranking models since users mainly care about the top ranked results in
real-world scenarios. A ranking model is said to be Certified Top-$K$ Robust on
a ranked list when it is guaranteed to keep documents that are out of the top
$K$ away from the top $K$ under any attack. Then, we introduce a Certified
Defense method, named CertDR, to achieve certified top-$K$ robustness against
WSRA, based on the idea of randomized smoothing. Specifically, we first
construct a smoothed ranker by applying random word substitutions on the
documents, and then leverage the ranking property jointly with the statistical
property of the ensemble to provably certify top-$K$ robustness. Extensive
experiments on two representative web search datasets demonstrate that CertDR
can significantly outperform state-of-the-art empirical defense methods for
ranking models.
