  A common retrieve-and-rerank paradigm involves retrieving a broad set of
relevant candidates using a scalable bi-encoder, followed by expensive but more
accurate cross-encoders to a limited candidate set. However, this small subset
often leads to error propagation from the bi-encoders, thereby restricting the
performance of the overall pipeline. To address these issues, we propose the
Comparing Multiple Candidates (CMC) framework, which compares a query and
multiple candidate embeddings jointly through shallow self-attention layers.
While providing contextualized representations, CMC is scalable enough to
handle multiple comparisons simultaneously, where comparing 2K candidates takes
only twice as long as comparing 100. Practitioners can use CMC as a lightweight
and effective reranker to improve top-1 accuracy. Moreover, when integrated
with another retriever, CMC reranking can function as a virtually enhanced
retriever. This configuration adds only negligible latency compared to using a
single retriever (virtual), while significantly improving recall at K
(enhanced).} Through experiments, we demonstrate that CMC, as a virtually
enhanced retriever, significantly improves Recall@k (+6.7, +3.5%-p for R@16,
R@64) compared to the initial retrieval stage on the ZeSHEL dataset. Meanwhile,
we conduct experiments for direct reranking on entity, passage, and dialogue
ranking. The results indicate that CMC is not only faster (11x) than
cross-encoders but also often more effective, with improved prediction
performance in Wikipedia entity linking (+0.7%-p) and DSTC7 dialogue ranking
(+3.3%-p). The code and link to datasets are available at
https://github.com/yc-song/cmc
