  Object manipulation is a critical skill required for Embodied AI agents
interacting with the world around them. Training agents to manipulate objects,
poses many challenges. These include occlusion of the target object by the
agent's arm, noisy object detection and localization, and the target frequently
going out of view as the agent moves around in the scene. We propose
Manipulation via Visual Object Location Estimation (m-VOLE), an approach that
explores the environment in search for target objects, computes their 3D
coordinates once they are located, and then continues to estimate their 3D
locations even when the objects are not visible, thus robustly aiding the task
of manipulating these objects throughout the episode. Our evaluations show a
massive 3x improvement in success rate over a model that has access to the same
sensory suite but is trained without the object location estimator, and our
analysis shows that our agent is robust to noise in depth perception and agent
localization. Importantly, our proposed approach relaxes several assumptions
about idealized localization and perception that are commonly employed by
recent works in embodied AI -- an important step towards training agents for
object manipulation in the real world.
