  Recent progress in information retrieval finds that embedding query and
document representation into multi-vector yields a robust bi-encoder retriever
on out-of-distribution datasets. In this paper, we explore whether late
interaction, the simplest form of multi-vector, is also helpful to neural
rerankers that only use the [CLS] vector to compute the similarity score.
Although intuitively, the attention mechanism of rerankers at the previous
layers already gathers the token-level information, we find adding late
interaction still brings an extra 5% improvement in average on
out-of-distribution datasets, with little increase in latency and no
degradation in in-domain effectiveness. Through extensive experiments and
analysis, we show that the finding is consistent across different model sizes
and first-stage retrievers of diverse natures and that the improvement is more
prominent on longer queries.
