  This paper presents a self-supervised method for learning reliable visual
correspondence from unlabeled videos. We formulate the correspondence as
finding paths in a joint space-time graph, where nodes are grid patches sampled
from frames, and are linked by two types of edges: (i) neighbor relations that
determine the aggregation strength from intra-frame neighbors in space, and
(ii) similarity relations that indicate the transition probability of
inter-frame paths across time. Leveraging the cycle-consistency in videos, our
contrastive learning objective discriminates dynamic objects from both their
neighboring views and temporal views. Compared with prior works, our approach
actively explores the neighbor relations of central instances to learn a latent
association between center-neighbor pairs (e.g., "hand -- arm") across time,
thus improving the instance discrimination. Without fine-tuning, our learned
representation outperforms the state-of-the-art self-supervised methods on a
variety of visual tasks including video object propagation, part propagation,
and pose keypoint tracking. Our self-supervised method also surpasses some
fully supervised algorithms designed for the specific tasks.
