  Continuous-time reinforcement learning tasks commonly use discrete steps of
fixed cycle times for actions. As practitioners need to choose the action-cycle
time for a given task, a significant concern is whether the hyper-parameters of
the learning algorithm need to be re-tuned for each choice of the cycle time,
which is prohibitive for real-world robotics. In this work, we investigate the
widely-used baseline hyper-parameter values of two policy gradient algorithms
-- PPO and SAC -- across different cycle times. Using a benchmark task where
the baseline hyper-parameters of both algorithms were shown to work well, we
reveal that when a cycle time different than the task default is chosen, PPO
with baseline hyper-parameters fails to learn. Moreover, both PPO and SAC with
their baseline hyper-parameters perform substantially worse than their tuned
values for each cycle time. We propose novel approaches for setting these
hyper-parameters based on the cycle time. In our experiments on simulated and
real-world robotic tasks, the proposed approaches performed at least as well as
the baseline hyper-parameters, with significantly better performance for most
choices of the cycle time, and did not result in learning failure for any cycle
time. Hyper-parameter tuning still remains a significant barrier for real-world
robotics, as our approaches require some initial tuning on a new task, even
though it is negligible compared to an extensive tuning for each cycle time.
Our approach requires no additional tuning after the cycle time is changed for
a given task and is a step toward avoiding extensive and costly hyper-parameter
tuning for real-world policy optimization.
