  We propose a general formulation for addressing reinforcement learning (RL)
problems in settings with observational data. That is, we consider the problem
of learning good policies solely from historical data in which unobserved
factors (confounders) affect both observed actions and rewards. Our formulation
allows us to extend a representative RL algorithm, the Actor-Critic method, to
its deconfounding variant, with the methodology for this extension being easily
applied to other RL algorithms. In addition to this, we develop a new benchmark
for evaluating deconfounding RL algorithms by modifying the OpenAI Gym
environments and the MNIST dataset. Using this benchmark, we demonstrate that
the proposed algorithms are superior to traditional RL methods in confounded
environments with observational data. To the best of our knowledge, this is the
first time that confounders are taken into consideration for addressing full RL
problems with observational data. Code is available at
https://github.com/CausalRL/DRL.
