  We study the problem of estimating the distribution of the return of a policy
using an offline dataset that is not generated from the policy, i.e.,
distributional offline policy evaluation (OPE). We propose an algorithm called
Fitted Likelihood Estimation (FLE), which conducts a sequence of Maximum
Likelihood Estimation (MLE) and has the flexibility of integrating any
state-of-the-art probabilistic generative models as long as it can be trained
via MLE. FLE can be used for both finite-horizon and infinite-horizon
discounted settings where rewards can be multi-dimensional vectors. Our
theoretical results show that for both finite-horizon and infinite-horizon
discounted settings, FLE can learn distributions that are close to the ground
truth under total variation distance and Wasserstein distance, respectively.
Our theoretical results hold under the conditions that the offline data covers
the test policy's traces and that the supervised learning MLE procedures
succeed. Experimentally, we demonstrate the performance of FLE with two
generative models, Gaussian mixture models and diffusion models. For the
multi-dimensional reward setting, FLE with diffusion models is capable of
estimating the complicated distribution of the return of a test policy.
