  Humans possess a versatile mechanism for extracting structured
representations of our visual world. When looking at an image, we can decompose
the scene into entities and their parts as well as obtain the dependencies
between them. To mimic such capability, we propose Visual Dependency
Transformers (DependencyViT) that can induce visual dependencies without any
labels. We achieve that with a novel neural operator called \emph{reversed
attention} that can naturally capture long-range visual dependencies between
image patches. Specifically, we formulate it as a dependency graph where a
child token in reversed attention is trained to attend to its parent tokens and
send information following a normalized probability distribution rather than
gathering information in conventional self-attention. With such a design,
hierarchies naturally emerge from reversed attention layers, and a dependency
tree is progressively induced from leaf nodes to the root node unsupervisedly.
  DependencyViT offers several appealing benefits. (i) Entities and their parts
in an image are represented by different subtrees, enabling part partitioning
from dependencies; (ii) Dynamic visual pooling is made possible. The leaf nodes
which rarely send messages can be pruned without hindering the model
performance, based on which we propose the lightweight DependencyViT-Lite to
reduce the computational and memory footprints; (iii) DependencyViT works well
on both self- and weakly-supervised pretraining paradigms on ImageNet, and
demonstrates its effectiveness on 8 datasets and 5 tasks, such as unsupervised
part and saliency segmentation, recognition, and detection.
