  Deep reinforcement learning has shown promising results on an abundance of
robotic tasks in simulation, including visual navigation and manipulation.
Prior work generally aims to build embodied agents that solve their assigned
tasks as quickly as possible, while largely ignoring the problems caused by
collision with objects during interaction. This lack of prioritization is
understandable: there is no inherent cost in breaking virtual objects. As a
result, "well-trained" agents frequently collide with objects before achieving
their primary goals, a behavior that would be catastrophic in the real world.
In this paper, we study the problem of training agents to complete the task of
visual mobile manipulation in the ManipulaTHOR environment while avoiding
unnecessary collision (disturbance) with objects. We formulate disturbance
avoidance as a penalty term in the reward function, but find that directly
training with such penalized rewards often results in agents being unable to
escape poor local optima. Instead, we propose a two-stage training curriculum
where an agent is first allowed to freely explore and build basic competencies
without penalization, after which a disturbance penalty is introduced to refine
the agent's behavior. Results on testing scenes show that our curriculum not
only avoids these poor local optima, but also leads to 10% absolute gains in
success rate without disturbance, compared to our state-of-the-art baselines.
Moreover, our curriculum is significantly more performant than a safe RL
algorithm that casts collision avoidance as a constraint. Finally, we propose a
novel disturbance-prediction auxiliary task that accelerates learning.
