  Query rewriting is a crucial technique for passage retrieval in open-domain
conversational question answering (CQA). It decontexualizes conversational
queries into self-contained questions suitable for off-the-shelf retrievers.
Existing methods attempt to incorporate retriever's preference during the
training of rewriting models. However, these approaches typically rely on
extensive annotations such as in-domain rewrites and/or relevant passage
labels, limiting the models' generalization and adaptation capabilities. In
this paper, we introduce AdaQR ($\textbf{Ada}$ptive $\textbf{Q}$uery
$\textbf{R}$ewriting), a framework for training query rewriting models with
limited rewrite annotations from seed datasets and completely no passage label.
Our approach begins by fine-tuning compact large language models using only
~$10\%$ of rewrite annotations from the seed dataset training split. The models
are then utilized to generate rewrite candidates for each query instance. A
novel approach is then proposed to assess retriever's preference for these
candidates by the probability of answers conditioned on the conversational
query by marginalizing the Top-$K$ passages. This serves as the reward for
optimizing the rewriter further using Direct Preference Optimization (DPO), a
process free of rewrite and retrieval annotations. Experimental results on four
open-domain CQA datasets demonstrate that AdaQR not only enhances the in-domain
capabilities of the rewriter with limited annotation requirement, but also
adapts effectively to out-of-domain datasets.
