  Learning continuous control in high-dimensional sparse reward settings, such
as robotic manipulation, is a challenging problem due to the number of samples
often required to obtain accurate optimal value and policy estimates. While
many deep reinforcement learning methods have aimed at improving sample
efficiency through replay or improved exploration techniques, state of the art
actor-critic and policy gradient methods still suffer from the hard exploration
problem in sparse reward settings. Motivated by recent successes of value-based
methods for approximating state-action values, like RBF-DQN, we explore the
potential of value-based reinforcement learning for learning continuous robotic
manipulation tasks in multi-task sparse reward settings. On robotic
manipulation tasks, we empirically show RBF-DQN converges faster than current
state of the art algorithms such as TD3, SAC, and PPO. We also perform ablation
studies with RBF-DQN and have shown that some enhancement techniques for
vanilla Deep Q learning such as Hindsight Experience Replay (HER) and
Prioritized Experience Replay (PER) can also be applied to RBF-DQN. Our
experimental analysis suggests that value-based approaches may be more
sensitive to data augmentation and replay buffer sample techniques than
policy-gradient methods, and that the benefits of these methods for robot
manipulation are heavily dependent on the transition dynamics of generated
subgoal states.
