  Pairing a lexical retriever with a neural re-ranking model has set
state-of-the-art performance on large-scale information retrieval datasets.
This pipeline covers scenarios like question answering or navigational queries,
however, for information-seeking scenarios, users often provide information on
whether a document is relevant to their query in form of clicks or explicit
feedback. Therefore, in this work, we explore how relevance feedback can be
directly integrated into neural re-ranking models by adopting few-shot and
parameter-efficient learning techniques. Specifically, we introduce a kNN
approach that re-ranks documents based on their similarity with the query and
the documents the user considers relevant. Further, we explore Cross-Encoder
models that we pre-train using meta-learning and subsequently fine-tune for
each query, training only on the feedback documents. To evaluate our different
integration strategies, we transform four existing information retrieval
datasets into the relevance feedback scenario. Extensive experiments
demonstrate that integrating relevance feedback directly in neural re-ranking
models improves their performance, and fusing lexical ranking with our best
performing neural re-ranker outperforms all other methods by 5.2 nDCG@20.
