  Using Reinforcement Learning (RL) in simulation to construct policies useful
in real life is challenging. This is often attributed to the sequential
decision making aspect: inaccuracies in simulation accumulate over multiple
steps, hence the simulated trajectories diverge from what would happen in
reality.
  In our work we show the need to consider another important aspect: the
mismatch in simulating control. We bring attention to the need for modeling
control as well as dynamics, since oversimplifying assumptions about applying
actions of RL policies could make the policies fail on real-world systems.
  We design a simulator for solving a pivoting task (of interest in Robotics)
and demonstrate that even a simple simulator designed with RL in mind
outperforms high-fidelity simulators when it comes to learning a policy that is
to be deployed on a real robotic system. We show that a phenomenon that is hard
to model - friction - could be exploited successfully, even when RL is
performed using a simulator with a simple dynamics and noise model. Hence, we
demonstrate that as long as the main sources of uncertainty are identified, it
could be possible to learn policies applicable to real systems even using a
simple simulator.
  RL-compatible simulators could open the possibilities for applying a wide
range of RL algorithms in various fields. This is important, since currently
data sparsity in fields like healthcare and education frequently forces
researchers and engineers to only consider sample-efficient RL approaches.
Successful simulator-aided RL could increase flexibility of experimenting with
RL algorithms and help applying RL policies to real-world settings in fields
where data is scarce. We believe that lessons learned in Robotics could help
other fields design RL-compatible simulators, so we summarize our experience
and conclude with suggestions.
