  Object grounding tasks aim to locate the target object in an image through
verbal communications. Understanding human command is an important process
needed for effective human-robot communication. However, this is challenging
because human commands can be ambiguous and erroneous. This paper aims to
disambiguate the human's referring expressions by allowing the agent to ask
relevant questions based on semantic data obtained from scene graphs. We test
if our agent can use relations between objects from a scene graph to ask
semantically relevant questions that can disambiguate the original user
command. In this paper, we present Incremental Grounding using Scene Graphs
(IGSG), a disambiguation model that uses semantic data from an image scene
graph and linguistic structures from a language scene graph to ground objects
based on human command. Compared to the baseline, IGSG shows promising results
in complex real-world scenes where there are multiple identical target objects.
IGSG can effectively disambiguate ambiguous or wrong referring expressions by
asking disambiguating questions back to the user.
