  Referring expressions usually describe an object using properties of the
object and relationships of the object with other objects. We propose a
technique that integrates context between objects to understand referring
expressions. Our approach uses an LSTM to learn the probability of a referring
expression, with input features from a region and a context region. The context
regions are discovered using multiple-instance learning (MIL) since annotations
for context objects are generally not available for training. We utilize
max-margin based MIL objective functions for training the LSTM. Experiments on
the Google RefExp and UNC RefExp datasets show that modeling context between
objects provides better performance than modeling only object properties. We
also qualitatively show that our technique can ground a referring expression to
its referred region along with the supporting context region.
