  Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a
specific image from a large gallery given a query sketch. Despite the
widespread applicability of FG-SBIR in many critical domains (e.g., crime
activity tracking), existing approaches still suffer from a low accuracy while
being sensitive to external noises such as unnecessary strokes in the sketch.
The retrieval performance will further deteriorate under a more practical
on-the-fly setting, where only a partially complete sketch with only a few
(noisy) strokes are available to retrieve corresponding images. We propose a
novel framework that leverages a uniquely designed deep reinforcement learning
model that performs a dual-level exploration to deal with partial sketch
training and attention region selection. By enforcing the model's attention on
the important regions of the original sketches, it remains robust to
unnecessary stroke noises and improve the retrieval accuracy by a large margin.
To sufficiently explore partial sketches and locate the important regions to
attend, the model performs bootstrapped policy gradient for global exploration
while adjusting a standard deviation term that governs a locator network for
local exploration. The training process is guided by a hybrid loss that
integrates a reinforcement loss and a supervised loss. A dynamic ranking reward
is developed to fit the on-the-fly image retrieval process using partial
sketches. The extensive experimentation performed on three public datasets
shows that our proposed approach achieves the state-of-the-art performance on
partial sketch based image retrieval.
