  Neural ranking models (NRMs) have shown remarkable success in recent years,
especially with pre-trained language models. However, deep neural models are
notorious for their vulnerability to adversarial examples. Adversarial attacks
may become a new type of web spamming technique given our increased reliance on
neural information retrieval models. Therefore, it is important to study
potential adversarial attacks to identify vulnerabilities of NRMs before they
are deployed. In this paper, we introduce the Word Substitution Ranking Attack
(WSRA) task against NRMs, which aims to promote a target document in rankings
by adding adversarial perturbations to its text. We focus on the decision-based
black-box attack setting, where the attackers cannot directly get access to the
model information, but can only query the target model to obtain the rank
positions of the partial retrieved list. This attack setting is realistic in
real-world search engines. We propose a novel Pseudo Relevance-based
ADversarial ranking Attack method (PRADA) that learns a surrogate model based
on Pseudo Relevance Feedback (PRF) to generate gradients for finding the
adversarial perturbations. Experiments on two web search benchmark datasets
show that PRADA can outperform existing attack strategies and successfully fool
the NRM with small indiscernible perturbations of text.
