  The scene graph generation (SGG) task aims to detect visual relationship
triplets, i.e., subject, predicate, object, in an image, providing a structural
vision layout for scene understanding. However, current models are stuck in
common predicates, e.g., "on" and "at", rather than informative ones, e.g.,
"standing on" and "looking at", resulting in the loss of precise information
and overall performance. If a model only uses "stone on road" rather than
"blocking" to describe an image, it is easy to misunderstand the scene. We
argue that this phenomenon is caused by two key imbalances between informative
predicates and common ones, i.e., semantic space level imbalance and training
sample level imbalance. To tackle this problem, we propose BA-SGG, a simple yet
effective SGG framework based on balance adjustment but not the conventional
distribution fitting. It integrates two components: Semantic Adjustment (SA)
and Balanced Predicate Learning (BPL), respectively for adjusting these
imbalances. Benefited from the model-agnostic process, our method is easily
applied to the state-of-the-art SGG models and significantly improves the SGG
performance. Our method achieves 14.3%, 8.0%, and 6.1% higher Mean Recall (mR)
than that of the Transformer model at three scene graph generation sub-tasks on
Visual Genome, respectively. Codes are publicly available.
