  Despite the impressive performance of recent unbiased Scene Graph Generation
(SGG) methods, the current debiasing literature mainly focuses on the
long-tailed distribution problem, whereas it overlooks another source of bias,
i.e., semantic confusion, which makes the SGG model prone to yield false
predictions for similar relationships. In this paper, we explore a debiasing
procedure for the SGG task leveraging causal inference. Our central insight is
that the Sparse Mechanism Shift (SMS) in causality allows independent
intervention on multiple biases, thereby potentially preserving head category
performance while pursuing the prediction of high-informative tail
relationships. However, the noisy datasets lead to unobserved confounders for
the SGG task, and thus the constructed causal models are always
causal-insufficient to benefit from SMS. To remedy this, we propose Two-stage
Causal Modeling (TsCM) for the SGG task, which takes the long-tailed
distribution and semantic confusion as confounders to the Structural Causal
Model (SCM) and then decouples the causal intervention into two stages. The
first stage is causal representation learning, where we use a novel Population
Loss (P-Loss) to intervene in the semantic confusion confounder. The second
stage introduces the Adaptive Logit Adjustment (AL-Adjustment) to eliminate the
long-tailed distribution confounder to complete causal calibration learning.
These two stages are model agnostic and thus can be used in any SGG model that
seeks unbiased predictions. Comprehensive experiments conducted on the popular
SGG backbones and benchmarks show that our TsCM can achieve state-of-the-art
performance in terms of mean recall rate. Furthermore, TsCM can maintain a
higher recall rate than other debiasing methods, which indicates that our
method can achieve a better tradeoff between head and tail relationships.
