  Existing sketch-analysis work studies sketches depicting static objects or
scenes. In this work, we propose a novel cross-modal retrieval problem of
fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a
sketch sequence is used as a query to retrieve a specific target video
instance. Compared with sketch-based still image retrieval, and coarse-grained
category-level video retrieval, this is more challenging as both visual
appearance and motion need to be simultaneously matched at a fine-grained
level. We contribute the first FG-SBVR dataset with rich annotations. We then
introduce a novel multi-stream multi-modality deep network to perform FG-SBVR
under both strong and weakly supervised settings. The key component of the
network is a relation module, designed to prevent model over-fitting given
scarce training data. We show that this model significantly outperforms a
number of existing state-of-the-art models designed for video analysis.
