  Recent developments of dense retrieval rely on quality representations of
queries and contexts from pre-trained query and context encoders. In this
paper, we introduce TOUR (Test-Time Optimization of Query Representations),
which further optimizes instance-level query representations guided by signals
from test-time retrieval results. We leverage a cross-encoder re-ranker to
provide fine-grained pseudo labels over retrieval results and iteratively
optimize query representations with gradient descent. Our theoretical analysis
reveals that TOUR can be viewed as a generalization of the classical Rocchio
algorithm for pseudo relevance feedback, and we present two variants that
leverage pseudo-labels as hard binary or soft continuous labels. We first apply
TOUR on phrase retrieval with our proposed phrase re-ranker, and also evaluate
its effectiveness on passage retrieval with an off-the-shelf re-ranker. TOUR
greatly improves end-to-end open-domain question answering accuracy, as well as
passage retrieval performance. TOUR also consistently improves direct
re-ranking by up to 2.0% while running 1.3-2.4x faster with an efficient
implementation.
