  Query expansion is a technique widely used in image search consisting in
combining highly ranked images from an original query into an expanded query
that is then reissued, generally leading to increased recall and precision. An
important aspect of query expansion is choosing an appropriate way to combine
the images into a new query. Interestingly, despite the undeniable empirical
success of query expansion, ad-hoc methods with different caveats have
dominated the landscape, and not a lot of research has been done on learning
how to do query expansion. In this paper we propose a more principled framework
to query expansion, where one trains, in a discriminative manner, a model that
learns how images should be aggregated to form the expanded query. Within this
framework, we propose a model that leverages a self-attention mechanism to
effectively learn how to transfer information between the different images
before aggregating them. Our approach obtains higher accuracy than existing
approaches on standard benchmarks. More importantly, our approach is the only
one that consistently shows high accuracy under different regimes, overcoming
caveats of existing methods.
