  Choosing an appropriate parameter set for the designed controller is critical
for the final performance but usually requires a tedious and careful tuning
process, which implies a strong need for automatic tuning methods. However,
among existing methods, derivative-free ones suffer from poor scalability or
low efficiency, while gradient-based ones are often unavailable due to possibly
non-differentiable controller structure. To resolve the issues, we tackle the
controller tuning problem using a novel derivative-free reinforcement learning
(RL) framework, which performs timestep-wise perturbation in parameter space
during experience collection and integrates derivative-free policy updates into
the advanced actor-critic RL architecture to achieve high versatility and
efficiency. To demonstrate the framework's efficacy, we conduct numerical
experiments on two concrete examples from autonomous driving, namely, adaptive
cruise control with PID controller and trajectory tracking with MPC controller.
Experimental results show that the proposed method outperforms popular
baselines and highlight its strong potential for controller tuning.
