  As a structured prediction task, scene graph generation, given an input
image, aims to explicitly model objects and their relationships by constructing
a visually-grounded scene graph. In the current literature, such task is
universally solved via a message passing neural network based mean field
variational Bayesian methodology. The classical loose evidence lower bound is
generally chosen as the variational inference objective, which could induce
oversimplified variational approximation and thus underestimate the underlying
complex posterior. In this paper, we propose a novel doubly reparameterized
importance weighted structure learning method, which employs a tighter
importance weighted lower bound as the variational inference objective. It is
computed from multiple samples drawn from a reparameterizable Gumbel-Softmax
sampler and the resulting constrained variational inference task is solved by a
generic entropic mirror descent algorithm. The resulting doubly reparameterized
gradient estimator reduces the variance of the corresponding derivatives with a
beneficial impact on learning. The proposed method achieves the
state-of-the-art performance on various popular scene graph generation
benchmarks.
