  Autonomous robotic systems, like autonomous vehicles and robotic search and
rescue, require efficient on-device training for continuous adaptation of Deep
Reinforcement Learning (DRL) models in dynamic environments. This research is
fundamentally motivated by the need to understand and address the challenges of
on-device real-time DRL, which involves balancing timing and algorithm
performance under memory constraints, as exposed through our extensive
empirical studies. This intricate balance requires co-optimizing two pivotal
parameters of DRL training -- batch size and replay buffer size. Configuring
these parameters significantly affects timing and algorithm performance, while
both (unfortunately) require substantial memory allocation to achieve
near-optimal performance.
  This paper presents R^3, a holistic solution for managing timing, memory, and
algorithm performance in on-device real-time DRL training. R^3 employs (i) a
deadline-driven feedback loop with dynamic batch sizing for optimizing timing,
(ii) efficient memory management to reduce memory footprint and allow larger
replay buffer sizes, and (iii) a runtime coordinator guided by heuristic
analysis and a runtime profiler for dynamically adjusting memory resource
reservations. These components collaboratively tackle the trade-offs in
on-device DRL training, improving timing and algorithm performance while
minimizing the risk of out-of-memory (OOM) errors.
  We implemented and evaluated R^3 extensively across various DRL frameworks
and benchmarks on three hardware platforms commonly adopted by autonomous
robotic systems. Additionally, we integrate R^3 with a popular realistic
autonomous car simulator to demonstrate its real-world applicability.
Evaluation results show that R^3 achieves efficacy across diverse platforms,
ensuring consistent latency performance and timing predictability with minimal
overhead.
