  Learning rewards from expert videos offers an affordable and effective
solution to specify the intended behaviors for reinforcement learning tasks. In
this work, we propose Diffusion Reward, a novel framework that learns rewards
from expert videos via conditional video diffusion models for solving complex
visual RL problems. Our key insight is that lower generative diversity is
observed when conditioned on expert trajectories. Diffusion Reward is
accordingly formalized by the negative of conditional entropy that encourages
productive exploration of expert-like behaviors. We show the efficacy of our
method over 10 robotic manipulation tasks from MetaWorld and Adroit with visual
input and sparse reward. Moreover, Diffusion Reward could even solve unseen
tasks successfully and effectively, largely surpassing baseline methods.
Project page and code: https://diffusion-reward.github.io/.
