  Visual grounding is a task to ground referring expressions in images, e.g.,
localize "the white truck in front of the yellow one". To resolve this task
fundamentally, the model should first find out the contextual objects (e.g.,
the "yellow" truck) and then exploit them to disambiguate the referent from
other similar objects by using the attributes and relationships (e.g., "white",
"yellow", "in front of"). However, due to the lack of annotations on contextual
objects and their relationships, existing methods degenerate the above joint
grounding process into a holistic association between the expression and
regions, thus suffering from unsatisfactory performance and limited
interpretability. In this paper, we alleviate the missing-annotation problem
and enable the joint reasoning by leveraging the language scene graph which
covers both labeled referent and unlabeled contexts (other objects, attributes,
and relationships). Specifically, the language scene graph is a graphical
representation where the nodes are objects with attributes and the edges are
relationships. We construct a factor graph based on it and then perform
marginalization over the graph, such that we can ground both referent and
contexts on corresponding image regions to achieve the joint visual grounding
(JVG). Experimental results demonstrate that the proposed approach is effective
and interpretable, e.g., on three benchmarks, it outperforms the
state-of-the-art methods while offers a complete grounding of all the objects
mentioned in the referring expression.
