  The recently released Ego4D dataset and benchmark significantly scales and
diversifies the first-person visual perception data. In Ego4D, the Visual
Queries 2D Localization task aims to retrieve objects appeared in the past from
the recording in the first-person view. This task requires a system to
spatially and temporally localize the most recent appearance of a given object
query, where query is registered by a single tight visual crop of the object in
a different scene.
  Our study is based on the three-stage baseline introduced in the Episodic
Memory benchmark. The baseline solves the problem by detection and tracking:
detect the similar objects in all the frames, then run a tracker from the most
confident detection result. In the VQ2D challenge, we identified two
limitations of the current baseline. (1) The training configuration has
redundant computation. Although the training set has millions of instances,
most of them are repetitive and the number of unique object is only around
14.6k. The repeated gradient computation of the same object lead to an
inefficient training; (2) The false positive rate is high on background frames.
This is due to the distribution gap between training and evaluation. During
training, the model is only able to see the clean, stable, and labeled frames,
but the egocentric videos also have noisy, blurry, or unlabeled background
frames. To this end, we developed a more efficient and effective solution.
Concretely, we bring the training loop from ~15 days to less than 24 hours, and
we achieve 0.17% spatial-temporal AP, which is 31% higher than the baseline.
Our solution got the first ranking on the public leaderboard. Our code is
publicly available at https://github.com/facebookresearch/vq2d_cvpr.
