  By assigning each relationship a single label, current approaches formulate
the relationship detection as a classification problem. Under this formulation,
predicate categories are treated as completely different classes. However,
different from the object labels where different classes have explicit
boundaries, predicates usually have overlaps in their semantic meanings. For
example, sit\_on and stand\_on have common meanings in vertical relationships
but different details of how these two objects are vertically placed. In order
to leverage the inherent structures of the predicate categories, we propose to
first build the language hierarchy and then utilize the Hierarchy Guided
Feature Learning (HGFL) strategy to learn better region features of both the
coarse-grained level and the fine-grained level. Besides, we also propose the
Hierarchy Guided Module (HGM) to utilize the coarse-grained level to guide the
learning of fine-grained level features. Experiments show that the proposed
simple yet effective method can improve several state-of-the-art baselines by a
large margin (up to $33\%$ relative gain) in terms of Recall@50 on the task of
Scene Graph Generation in different datasets.
