  Effective decision making involves flexibly relating past experiences and
relevant contextual information to a novel situation. In deep reinforcement
learning (RL), the dominant paradigm is for an agent to amortise information
that helps decision making into its network weights via gradient descent on
training losses. Here, we pursue an alternative approach in which agents can
utilise large-scale context sensitive database lookups to support their
parametric computations. This allows agents to directly learn in an end-to-end
manner to utilise relevant information to inform their outputs. In addition,
new information can be attended to by the agent, without retraining, by simply
augmenting the retrieval dataset. We study this approach for offline RL in 9x9
Go, a challenging game for which the vast combinatorial state space privileges
generalisation over direct matching to past experiences. We leverage fast,
approximate nearest neighbor techniques in order to retrieve relevant data from
a set of tens of millions of expert demonstration states. Attending to this
information provides a significant boost to prediction accuracy and game-play
performance over simply using these demonstrations as training trajectories,
providing a compelling demonstration of the value of large-scale retrieval in
offline RL agents.
