  Dual-encoder-based dense retrieval models have become the standard in IR.
They employ large Transformer-based language models, which are notoriously
inefficient in terms of resources and latency. We propose Fast-Forward indexes
-- vector forward indexes which exploit the semantic matching capabilities of
dual-encoder models for efficient and effective re-ranking. Our framework
enables re-ranking at very high retrieval depths and combines the merits of
both lexical and semantic matching via score interpolation. Furthermore, in
order to mitigate the limitations of dual-encoders, we tackle two main
challenges: Firstly, we improve computational efficiency by either
pre-computing representations, avoiding unnecessary computations altogether, or
reducing the complexity of encoders. This allows us to considerably improve
ranking efficiency and latency. Secondly, we optimize the memory footprint and
maintenance cost of indexes; we propose two complementary techniques to reduce
the index size and show that, by dynamically dropping irrelevant document
tokens, the index maintenance efficiency can be improved substantially. We
perform evaluation to show the effectiveness and efficiency of Fast-Forward
indexes -- our method has low latency and achieves competitive results without
the need for hardware acceleration, such as GPUs.
