  General contact-rich manipulation problems are long-standing challenges in
robotics due to the difficulty of understanding complicated contact physics.
Deep reinforcement learning (RL) has shown great potential in solving robot
manipulation tasks. However, existing RL policies have limited adaptability to
environments with diverse dynamics properties, which is pivotal in solving many
contact-rich manipulation tasks. In this work, we propose Contact-aware Online
COntext Inference (COCOI), a deep RL method that encodes a context embedding of
dynamics properties online using contact-rich interactions. We study this
method based on a novel and challenging non-planar pushing task, where the
robot uses a monocular camera image and wrist force torque sensor reading to
push an object to a goal location while keeping it upright. We run extensive
experiments to demonstrate the capability of COCOI in a wide range of settings
and dynamics properties in simulation, and also in a sim-to-real transfer
scenario on a real robot (Video: https://youtu.be/nrmJYksh1Kc)
