  We propose a novel iterative approach for crossing the reality gap that
utilises live robot rollouts and differentiable physics. Our method,
RealityGrad, demonstrates for the first time, an efficient sim2real transfer in
combination with a real2sim model optimisation for closing the reality gap.
Differentiable physics has become an alluring alternative to classical
rigid-body simulation due to the current culmination of automatic
differentiation libraries, compute and non-linear optimisation libraries. Our
method builds on this progress and employs differentiable physics for efficient
trajectory optimisation. We demonstrate RealitGrad on a dynamic control task
for a serial link robot manipulator and present results that show its
efficiency and ability to quickly improve not just the robot's performance in
real world tasks but also enhance the simulation model for future tasks. One
iteration of RealityGrad takes less than 22 minutes on a desktop computer while
reducing the error by 2/3, making it efficient compared to other sim2real
methods in both compute and time. Our methodology and application of
differentiable physics establishes a promising approach for crossing the
reality gap and has great potential for scaling to complex environments.
