  Diffusion models have seen tremendous success as generative architectures.
Recently, they have been shown to be effective at modelling policies for
offline reinforcement learning and imitation learning. We explore using
diffusion as a model class for the successor state measure (SSM) of a policy.
We find that enforcing the Bellman flow constraints leads to a simple Bellman
update on the diffusion step distribution.
