  We propose Adversarially Trained Actor Critic (ATAC), a new model-free
algorithm for offline reinforcement learning (RL) under insufficient data
coverage, based on the concept of relative pessimism. ATAC is designed as a
two-player Stackelberg game: A policy actor competes against an adversarially
trained value critic, who finds data-consistent scenarios where the actor is
inferior to the data-collection behavior policy. We prove that, when the actor
attains no regret in the two-player game, running ATAC produces a policy that
provably 1) outperforms the behavior policy over a wide range of
hyperparameters that control the degree of pessimism, and 2) competes with the
best policy covered by data with appropriately chosen hyperparameters. Compared
with existing works, notably our framework offers both theoretical guarantees
for general function approximation and a deep RL implementation scalable to
complex environments and large datasets. In the D4RL benchmark, ATAC
consistently outperforms state-of-the-art offline RL algorithms on a range of
continuous control tasks.
