  A popular paradigm for offline Reinforcement Learning (RL) tasks is to first
fit the offline trajectories to a sequence model, and then prompt the model for
actions that lead to high expected return. In addition to obtaining accurate
sequence models, this paper highlights that tractability, the ability to
exactly and efficiently answer various probabilistic queries, plays an
important role in offline RL. Specifically, due to the fundamental
stochasticity from the offline data-collection policies and the environment
dynamics, highly non-trivial conditional/constrained generation is required to
elicit rewarding actions. it is still possible to approximate such queries, we
observe that such crude estimates significantly undermine the benefits brought
by expressive sequence models. To overcome this problem, this paper proposes
Trifle (Tractable Inference for Offline RL), which leverages modern Tractable
Probabilistic Models (TPMs) to bridge the gap between good sequence models and
high expected returns at evaluation time. Empirically, Trifle achieves the most
state-of-the-art scores in 9 Gym-MuJoCo benchmarks against strong baselines.
Further, owing to its tractability, Trifle significantly outperforms prior
approaches in stochastic environments and safe RL tasks (e.g. with action
constraints) with minimum algorithmic modifications.
