  In modern robotics, effectively computing optimal control policies under
dynamically varying environments poses substantial challenges to the
off-the-shelf parametric policy gradient methods, such as the Deep
Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic policy
gradient (TD3). In this paper, we propose a systematic methodology to
dynamically learn a sequence of optimal control policies non-parametrically,
while autonomously adapting with the constantly changing environment dynamics.
Specifically, our non-parametric kernel-based methodology embeds a policy
distribution as the features in a non-decreasing Euclidean space, therefore
allowing its search space to be defined as a very high (possible infinite)
dimensional RKHS (Reproducing Kernel Hilbert Space). Moreover, by leveraging
the similarity metric computed in RKHS, we augmented our non-parametric
learning with the technique of AdaptiveH- adaptively selecting a time-frame
window of finishing the optimal part of whole action-sequence sampled on some
preceding observed state. To validate our proposed approach, we conducted
extensive experiments with multiple classic benchmarks and one simulated
robotics benchmark equipped with dynamically changing environments. Overall,
our methodology has outperformed the well-established DDPG and TD3 methodology
by a sizeable margin in terms of learning performance.
