  Access to online visual search engines implies sharing of private user
content - the query images. We introduce the concept of targeted mismatch
attack for deep learning based retrieval systems to generate an adversarial
image to conceal the query image. The generated image looks nothing like the
user intended query, but leads to identical or very similar retrieval results.
Transferring attacks to fully unseen networks is challenging. We show
successful attacks to partially unknown systems, by designing various loss
functions for the adversarial image construction. These include loss functions,
for example, for unknown global pooling operation or unknown input resolution
by the retrieval system. We evaluate the attacks on standard retrieval
benchmarks and compare the results retrieved with the original and adversarial
image.
