  Most existing image retrieval systems use text queries as a way for the user
to express what they are looking for. However, fine-grained image retrieval
often requires the ability to also express where in the image the content they
are looking for is. The text modality can only cumbersomely express such
localization preferences, whereas pointing is a more natural fit. In this
paper, we propose an image retrieval setup with a new form of multimodal
queries, where the user simultaneously uses both spoken natural language (the
what) and mouse traces over an empty canvas (the where) to express the
characteristics of the desired target image. We then describe simple
modifications to an existing image retrieval model, enabling it to operate in
this setup. Qualitative and quantitative experiments show that our model
effectively takes this spatial guidance into account, and provides
significantly more accurate retrieval results compared to text-only equivalent
systems.
