  Modern sequence-to-sequence relevance models like monoT5 can effectively
capture complex textual interactions between queries and documents through
cross-encoding. However, the use of natural language tokens in prompts, such as
Query, Document, and Relevant for monoT5, opens an attack vector for malicious
documents to manipulate their relevance score through prompt injection, e.g.,
by adding target words such as true. Since such possibilities have not yet been
considered in retrieval evaluation, we analyze the impact of query-independent
prompt injection via manually constructed templates and LLM-based rewriting of
documents on several existing relevance models. Our experiments on the TREC
Deep Learning track show that adversarial documents can easily manipulate
different sequence-to-sequence relevance models, while BM25 (as a typical
lexical model) is not affected. Remarkably, the attacks also affect
encoder-only relevance models (which do not rely on natural language prompt
tokens), albeit to a lesser extent.
