  Deep Reinforcement Learning (DRL) has achieved remarkable success, ranging
from complex computer games to real-world applications, showing the potential
for intelligent agents capable of learning in dynamic environments. However,
its application in real-world scenarios presents challenges, including the
jerky problem, in which jerky trajectories not only compromise system safety
but also increase power consumption and shorten the service life of robotic and
autonomous systems. To address jerky actions, a method called conditioning for
action policy smoothness (CAPS) was proposed by adding regularization terms to
reduce the action changes. This paper further proposes a novel method, named
Gradient-based CAPS (Grad-CAPS), that modifies CAPS by reducing the difference
in the gradient of action and then uses displacement normalization to enable
the agent to adapt to invariant action scales. Consequently, our method
effectively reduces zigzagging action sequences while enhancing policy
expressiveness and the adaptability of our method across diverse scenarios and
environments. In the experiments, we integrated Grad-CAPS with different
reinforcement learning algorithms and evaluated its performance on various
robotic-related tasks in DeepMind Control Suite and OpenAI Gym environments.
The results demonstrate that Grad-CAPS effectively improves performance while
maintaining a comparable level of smoothness compared to CAPS and Vanilla
agents.
