  We propose Ephemeral Value Adjusments (EVA): a means of allowing deep
reinforcement learning agents to rapidly adapt to experience in their replay
buffer. EVA shifts the value predicted by a neural network with an estimate of
the value function found by planning over experience tuples from the replay
buffer near the current state. EVA combines a number of recent ideas around
combining episodic memory-like structures into reinforcement learning agents:
slot-based storage, content-based retrieval, and memory-based planning. We show
that EVAis performant on a demonstration task and Atari games.
