  With rising uncertainty in the real world, online Reinforcement Learning (RL)
has been receiving increasing attention due to its fast learning capability and
improving data efficiency. However, online RL often suffers from complex Value
Function Approximation (VFA) and catastrophic interference, creating difficulty
for the deep neural network to be applied to an online RL algorithm in a fully
online setting. Therefore, a simpler and more adaptive approach is introduced
to evaluate value function with the kernel-based model. Sparse representations
are superior at handling interference, indicating that competitive sparse
representations should be learnable, non-prior, non-truncated and explicit when
compared with current sparse representation methods. Moreover, in learning
sparse representations, attention mechanisms are utilized to represent the
degree of sparsification, and a smooth attentive function is introduced into
the kernel-based VFA. In this paper, we propose an Online Attentive
Kernel-Based Temporal Difference (OAKTD) algorithm using two-timescale
optimization and provide convergence analysis of our proposed algorithm.
Experimental evaluations showed that OAKTD outperformed several Online
Kernel-based Temporal Difference (OKTD) learning algorithms in addition to the
Temporal Difference (TD) learning algorithm with Tile Coding on public Mountain
Car, Acrobot, CartPole and Puddle World tasks.
