  Incorporating relational reasoning in neural networks for object recognition
remains an open problem. Although many attempts have been made for relational
reasoning, they generally only consider a single type of relationship. For
example, pixel relations through self-attention (e.g., non-local networks),
scale relations through feature fusion (e.g., feature pyramid networks), or
object relations through graph convolutions (e.g., reasoning-RCNN). Little
attention has been given to more generalized frameworks that can reason across
these relationships. In this paper, we propose a hierarchical relational
reasoning framework (HR-RCNN) for object detection, which utilizes a novel
graph attention module (GAM). This GAM is a concise module that enables
reasoning across heterogeneous nodes by operating on the graph edges directly.
Leveraging heterogeneous relationships, our HR-RCNN shows great improvement on
COCO dataset, for both object detection and instance segmentation.
