  We tackle the problem of visual search under resource constraints. Existing
systems use the same embedding model to compute representations (embeddings)
for the query and gallery images. Such systems inherently face a hard
accuracy-efficiency trade-off: the embedding model needs to be large enough to
ensure high accuracy, yet small enough to enable query-embedding computation on
resource-constrained platforms. This trade-off could be mitigated if gallery
embeddings are generated from a large model and query embeddings are extracted
using a compact model. The key to building such a system is to ensure
representation compatibility between the query and gallery models. In this
paper, we address two forms of compatibility: One enforced by modifying the
parameters of each model that computes the embeddings. The other by modifying
the architectures that compute the embeddings, leading to compatibility-aware
neural architecture search (CMP-NAS). We test CMP-NAS on challenging retrieval
tasks for fashion images (DeepFashion2), and face images (IJB-C). Compared to
ordinary (homogeneous) visual search using the largest embedding model
(paragon), CMP-NAS achieves 80-fold and 23-fold cost reduction while
maintaining accuracy within 0.3% and 1.6% of the paragon on DeepFashion2 and
IJB-C respectively.
