  Deep Reinforcement Learning (RL) can yield capable agents and control
policies in several domains but is commonly plagued by prohibitively long
training times. Additionally, in the case of continuous control problems, the
applicability of learned policies on real-world embedded devices is limited due
to the lack of real-time guarantees and portability of existing libraries. To
address these challenges, we present RLtools, a dependency-free, header-only,
pure C++ library for deep supervised and reinforcement learning. Its novel
architecture allows RLtools to be used on a wide variety of platforms, from HPC
clusters over workstations and laptops to smartphones, smartwatches, and
microcontrollers. Specifically, due to the tight integration of the RL
algorithms with simulation environments, RLtools can solve popular RL problems
up to 76 times faster than other popular RL frameworks. We also benchmark the
inference on a diverse set of microcontrollers and show that in most cases our
optimized implementation is by far the fastest. Finally, RLtools enables the
first-ever demonstration of training a deep RL algorithm directly on a
microcontroller, giving rise to the field of Tiny Reinforcement Learning
(TinyRL). The source code as well as documentation and live demos are available
through our project page at https://rl.tools.
