  This paper studies visual search using structured queries. The structure is
in the form of a 2D composition that encodes the position and the category of
the objects. The transformation of the position and the category of the objects
leads to a continuous-valued relationship between visual compositions, which
carries highly beneficial information, although not leveraged by previous
techniques. To that end, in this work, our goal is to leverage these continuous
relationships by using the notion of symmetry in equivariance. Our model output
is trained to change symmetrically with respect to the input transformations,
leading to a sensitive feature space. Doing so leads to a highly efficient
search technique, as our approach learns from fewer data using a smaller
feature space. Experiments on two large-scale benchmarks of MS-COCO and
HICO-DET demonstrates that our approach leads to a considerable gain in the
performance against competing techniques.
