  Learning robotic control policies in the real world gives rise to challenges
in data efficiency, safety, and controlling the initial condition of the
system. On the other hand, simulations are a useful alternative as they provide
an abundant source of data without the restrictions of the real world.
Unfortunately, simulations often fail to accurately model complex real-world
phenomena. Traditional system identification techniques are limited in
expressiveness by the analytical model parameters, and usually are not
sufficient to capture such phenomena. In this paper we propose a general
framework for improving the analytical model by optimizing state dependent
generalized forces. State dependent generalized forces are expressive enough to
model constraints in the equations of motion, while maintaining a clear
physical meaning and intuition. We use reinforcement learning to efficiently
optimize the mapping from states to generalized forces over a discounted
infinite horizon. We show that using only minutes of real world data improves
the sim-to-real control policy transfer. We demonstrate the feasibility of our
approach by validating it on a nonprehensile manipulation task on the Sawyer
robot.
