  Adversarial examples, generated by applying small perturbations to input
features, are widely used to fool classifiers and measure their robustness to
noisy inputs. However, little work has been done to evaluate the robustness of
ranking models through adversarial examples. In this work, we present a
systematic approach of leveraging adversarial examples to measure the
robustness of popular ranking models. We explore a simple method to generate
adversarial examples that forces a ranker to incorrectly rank the documents.
Using this approach, we analyze the robustness of various ranking models and
the quality of perturbations generated by the adversarial attacker across two
datasets. Our findings suggest that with very few token changes (1-3), the
attacker can yield semantically similar perturbed documents that can fool
different rankers into changing a document's score, lowering its rank by
several positions.
