  We propose a novel framework for efficient parallelization of deep
reinforcement learning algorithms, enabling these algorithms to learn from
multiple actors on a single machine. The framework is algorithm agnostic and
can be applied to on-policy, off-policy, value based and policy gradient based
algorithms. Given its inherent parallelism, the framework can be efficiently
implemented on a GPU, allowing the usage of powerful models while significantly
reducing training time. We demonstrate the effectiveness of our framework by
implementing an advantage actor-critic algorithm on a GPU, using on-policy
experiences and employing synchronous updates. Our algorithm achieves
state-of-the-art performance on the Atari domain after only a few hours of
training. Our framework thus opens the door for much faster experimentation on
demanding problem domains. Our implementation is open-source and is made public
at https://github.com/alfredvc/paac
