  Reliant on too many experiments to learn good actions, current Reinforcement
Learning (RL) algorithms have limited applicability in real-world settings,
which can be too expensive to allow exploration. We propose an algorithm for
batch RL, where effective policies are learned using only a fixed offline
dataset instead of online interactions with the environment. The limited data
in batch RL produces inherent uncertainty in value estimates of states/actions
that were insufficiently represented in the training data. This leads to
particularly severe extrapolation when our candidate policies diverge from one
that generated the data. We propose to mitigate this issue via two
straightforward penalties: a policy-constraint to reduce this divergence and a
value-constraint that discourages overly optimistic estimates. Over a
comprehensive set of 32 continuous-action batch RL benchmarks, our approach
compares favorably to state-of-the-art methods, regardless of how the offline
data were collected.
