  Deep reinforcement learning (RL) has made groundbreaking advancements in
robotics, data center management and other applications. Unfortunately,
system-level bottlenecks in RL workloads are poorly understood; we observe
fundamental structural differences in RL workloads that make them inherently
less GPU-bound than supervised learning (SL). To explain where training time is
spent in RL workloads, we propose RL-Scope, a cross-stack profiler that scopes
low-level CPU/GPU resource usage to high-level algorithmic operations, and
provides accurate insights by correcting for profiling overhead. Using
RL-Scope, we survey RL workloads across its major dimensions including ML
backend, RL algorithm, and simulator. For ML backends, we explain a $2.3\times$
difference in runtime between equivalent PyTorch and TensorFlow algorithm
implementations, and identify a bottleneck rooted in overly abstracted
algorithm implementations. For RL algorithms and simulators, we show that
on-policy algorithms are at least $3.5\times$ more simulation-bound than
off-policy algorithms. Finally, we profile a scale-up workload and demonstrate
that GPU utilization metrics reported by commonly used tools dramatically
inflate GPU usage, whereas RL-Scope reports true GPU-bound time. RL-Scope is an
open-source tool available at https://github.com/UofT-EcoSystem/rlscope .
