  Model-based next state prediction and state value prediction are slow to
converge. To address these challenges, we do the following: i) Instead of a
neural network, we do model-based planning using a parallel memory retrieval
system (which we term the slow mechanism); ii) Instead of learning state
values, we guide the agent's actions using goal-directed exploration, by using
a neural network to choose the next action given the current state and the goal
state (which we term the fast mechanism). The goal-directed exploration is
trained online using hippocampal replay of visited states and future imagined
states every single time step, leading to fast and efficient training.
Empirical studies show that our proposed method has a 92% solve rate across 100
episodes in a dynamically changing grid world, significantly outperforming
state-of-the-art actor critic mechanisms such as PPO (54%), TRPO (50%) and A2C
(24%). Ablation studies demonstrate that both mechanisms are crucial. We posit
that the future of Reinforcement Learning (RL) will be to model goals and
sub-goals for various tasks, and plan it out in a goal-directed memory-based
approach.
