  The offline reinforcement learning (RL) problem is often motivated by the
need to learn data-driven decision policies in financial, legal and healthcare
applications. However, the learned policy could retain sensitive information of
individuals in the training data (e.g., treatment and outcome of patients),
thus susceptible to various privacy risks. We design offline RL algorithms with
differential privacy guarantees which provably prevent such risks. These
algorithms also enjoy strong instance-dependent learning bounds under both
tabular and linear Markov decision process (MDP) settings. Our theory and
simulation suggest that the privacy guarantee comes at (almost) no drop in
utility comparing to the non-private counterpart for a medium-size dataset.
