  Neural ranking models (NRMs) have undergone significant development and have
become integral components of information retrieval (IR) systems.
Unfortunately, recent research has unveiled the vulnerability of NRMs to
adversarial document manipulations, potentially exploited by malicious search
engine optimization practitioners. While progress in adversarial attack
strategies aids in identifying the potential weaknesses of NRMs before their
deployment, the defensive measures against such attacks, like the detection of
adversarial documents, remain inadequately explored. To mitigate this gap, this
paper establishes a benchmark dataset to facilitate the investigation of
adversarial ranking defense and introduces two types of detection tasks for
adversarial documents. A comprehensive investigation of the performance of
several detection baselines is conducted, which involve examining the
spamicity, perplexity, and linguistic acceptability, and utilizing supervised
classifiers. Experimental results demonstrate that a supervised classifier can
effectively mitigate known attacks, but it performs poorly against unseen
attacks. Furthermore, such classifier should avoid using query text to prevent
learning the classification on relevance, as it might lead to the inadvertent
discarding of relevant documents.
