  Instance retrieval requires one to search for images that contain a
particular object within a large corpus. Recent studies show that using image
features generated by pooling convolutional layer feature maps (CFMs) of a
pretrained convolutional neural network (CNN) leads to promising performance
for this task. However, due to the global pooling strategy adopted in those
works, the generated image feature is less robust to image clutter and tends to
be contaminated by the irrelevant image patterns. In this article, we alleviate
this drawback by proposing a novel reranking algorithm using CFMs to refine the
retrieval result obtained by existing methods. Our key idea, called query
adaptive matching (QAM), is to first represent the CFMs of each image by a set
of base regions which can be freely combined into larger regions-of-interest.
Then the similarity between the query and a candidate image is measured by the
best similarity score that can be attained by comparing the query feature and
the feature pooled from a combined region. We show that the above procedure can
be cast as an optimization problem and it can be solved efficiently with an
off-the-shelf solver. Besides this general framework, we also propose two
practical ways to create the base regions. One is based on the property of the
CFM and the other one is based on a multi-scale spatial pyramid scheme. Through
extensive experiments, we show that our reranking approaches bring substantial
performance improvement and by applying them we can outperform the state of the
art on several instance retrieval benchmarks.
