  Panoptic Scene Graph generation (PSG) is a recently proposed task in image
scene understanding that aims to segment the image and extract triplets of
subjects, objects and their relations to build a scene graph. This task is
particularly challenging for two reasons. First, it suffers from a long-tail
problem in its relation categories, making naive biased methods more inclined
to high-frequency relations. Existing unbiased methods tackle the long-tail
problem by data/loss rebalancing to favor low-frequency relations. Second, a
subject-object pair can have two or more semantically overlapping relations.
While existing methods favor one over the other, our proposed HiLo framework
lets different network branches specialize on low and high frequency relations,
enforce their consistency and fuse the results. To the best of our knowledge we
are the first to propose an explicitly unbiased PSG method. In extensive
experiments we show that our HiLo framework achieves state-of-the-art results
on the PSG task. We also apply our method to the Scene Graph Generation task
that predicts boxes instead of masks and see improvements over all baseline
methods. Code is available at https://github.com/franciszzj/HiLo.
