  A core operation in reinforcement learning (RL) is finding an action that is
optimal with respect to a learned value function. This operation is often
challenging when the learned value function takes continuous actions as input.
We introduce deep radial-basis value functions (RBVFs): value functions learned
using a deep network with a radial-basis function (RBF) output layer. We show
that the maximum action-value with respect to a deep RBVF can be approximated
easily and accurately. Moreover, deep RBVFs can represent any true value
function owing to their support for universal function approximation. We extend
the standard DQN algorithm to continuous control by endowing the agent with a
deep RBVF. We show that the resultant agent, called RBF-DQN, significantly
outperforms value-function-only baselines, and is competitive with
state-of-the-art actor-critic algorithms.
