  Simulation is essential to reinforcement learning (RL) before implementation
in the real world, especially for safety-critical applications like robot
manipulation. Conventionally, RL agents are sensitive to the discrepancies
between the simulation and the real world, known as the sim-to-real gap. The
application of domain randomization, a technique used to fill this gap, is
limited to the imposition of heuristic-randomized models. {We investigate the
properties of intrinsic stochasticity of real-time simulation (RT-IS) of
off-the-shelf simulation software and its potential to improve RL performance.
This improvement includes a higher tolerance to noise and model imprecision and
superiority to conventional domain randomization in terms of ease of use and
automation. Firstly, we conduct analytical studies to measure the correlation
of RT-IS with the utilization of computer hardware and validate its
comparability with the natural stochasticity of a physical robot. Then, we
exploit the RT-IS feature in the training of an RL agent. The simulation and
physical experiment results verify the feasibility and applicability of RT-IS
to robust agent training for robot manipulation tasks. The RT-IS-powered RL
agent outperforms conventional agents on robots with modeling uncertainties.
RT-IS requires less heuristic randomization, is not task-dependent, and
achieves better generalizability than the conventional
domain-randomization-powered agents. Our findings provide a new perspective on
the sim-to-real problem in practical applications like robot manipulation
tasks.
