  Conventional phrase grounding aims to localize noun phrases mentioned in a
given caption to their corresponding image regions, which has achieved great
success recently. Apparently, sole noun phrase grounding is not enough for
cross-modal visual language understanding. Here we extend the task by
considering pronouns as well. First, we construct a dataset of phrase grounding
with both noun phrases and pronouns to image regions. Based on the dataset, we
test the performance of phrase grounding by using a state-of-the-art literature
model of this line. Then, we enhance the baseline grounding model with
coreference information which should help our task potentially, modeling the
coreference structures with graph convolutional networks. Experiments on our
dataset, interestingly, show that pronouns are easier to ground than noun
phrases, where the possible reason might be that these pronouns are much less
ambiguous. Additionally, our final model with coreference information can
significantly boost the grounding performance of both noun phrases and
pronouns.
