  Grounded understanding of natural language in physical scenes can greatly
benefit robots that follow human instructions. In object manipulation
scenarios, existing end-to-end models are proficient at understanding semantic
concepts, but typically cannot handle complex instructions involving spatial
relations among multiple objects. which require both reasoning object-level
spatial relations and learning precise pixel-level manipulation affordances. We
take an initial step to this challenge with a decoupled two-stage solution. In
the first stage, we propose an object-centric semantic-spatial reasoner to
select which objects are relevant for the language instructed task. The
segmentation of selected objects are then fused as additional input to the
affordance learning stage. Simply incorporating the inductive bias of relevant
objects to a vision-language affordance learning agent can effectively boost
its performance in a custom testbed designed for object manipulation with
spatial-related language instructions.
