  Deep generative models (DGMs) have demonstrated great success across various
domains, particularly in generating texts, images, and videos using models
trained from offline data. Similarly, data-driven decision-making and robotic
control also necessitate learning a generator function from the offline data to
serve as the strategy or policy. In this case, applying deep generative models
in offline policy learning exhibits great potential, and numerous studies have
explored in this direction. However, this field still lacks a comprehensive
review and so developments of different branches are relatively independent. In
this paper, we provide the first systematic review on the applications of deep
generative models for offline policy learning. In particular, we cover five
mainstream deep generative models, including Variational Auto-Encoders,
Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion
Models, and their applications in both offline reinforcement learning (offline
RL) and imitation learning (IL). Offline RL and IL are two main branches of
offline policy learning and are widely-adopted techniques for sequential
decision-making. Notably, for each type of DGM-based offline policy learning,
we distill its fundamental scheme, categorize related works based on the usage
of the DGM, and sort out the development process of algorithms in that field.
Subsequent to the main content, we provide in-depth discussions on deep
generative models and offline policy learning as a summary, based on which we
present our perspectives on future research directions. This work offers a
hands-on reference for the research progress in deep generative models for
offline policy learning, and aims to inspire improved DGM-based offline RL or
IL algorithms. For convenience, we maintain a paper list on
https://github.com/LucasCJYSDL/DGMs-for-Offline-Policy-Learning.
