  Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively
large number of training samples for learning complex tasks. Many recent works
on speeding up Deep RL have focused on distributed training and simulation.
While distributed training is often done on the GPU, simulation is not. In this
work, we propose using GPU-accelerated RL simulations as an alternative to CPU
ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising
speed-ups of learning various continuous-control, locomotion tasks. With one
GPU and CPU core, we are able to train the Humanoid running task in less than
20 minutes, using 10-1000x fewer CPU cores than previous works. We also
demonstrate the scalability of our simulator to multi-GPU settings to train
more challenging locomotion tasks.
