  In state of the art model-free off-policy deep reinforcement learning, a
replay memory is used to store past experience and derive all network updates.
Even if both state and action spaces are continuous, the replay memory only
holds a finite number of transitions. We represent these transitions in a data
graph and link its structure to soft divergence. By selecting a subgraph with a
favorable structure, we construct a simplified Markov Decision Process for
which exact Q-values can be computed efficiently as more data comes in. The
subgraph and its associated Q-values can be represented as a QGraph. We show
that the Q-value for each transition in the simplified MDP is a lower bound of
the Q-value for the same transition in the original continuous Q-learning
problem. By using these lower bounds in temporal difference learning, our
method QG-DDPG is less prone to soft divergence and exhibits increased sample
efficiency while being more robust to hyperparameters. QGraphs also retain
information from transitions that have already been overwritten in the replay
memory, which can decrease the algorithm's sensitivity to the replay memory
capacity.
