# Value-guided planning

> [!WARNING]
> 🧪 This is an experimental pipeline for reinforcement learning!

This pipeline is based on the [Planning with Diffusion for Flexible Behavior Synthesis](https://huggingface.co/papers/2205.09991) paper by Michael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey Levine.

The abstract from the paper is:

*Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers. While conceptually simple, this combination has a number of empirical shortcomings, suggesting that learned models may not be well-suited to standard trajectory optimization. In this paper, we consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem, such that sampling from the model and planning with it become nearly identical. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories. We show how classifier-guided sampling and image inpainting can be reinterpreted as coherent planning strategies, explore the unusual and useful properties of diffusion-based planning methods, and demonstrate the effectiveness of our framework in control settings that emphasize long-horizon decision-making and test-time flexibility.*

You can find additional information about the model on the [project page](https://diffusion-planning.github.io/), the [original codebase](https://github.com/jannerm/diffuser), or try it out in a demo [notebook](https://colab.research.google.com/drive/1rXm8CX4ZdN5qivjJ2lhwhkOmt_m0CvU0#scrollTo=6HXJvhyqcITc&uniqifier=1).

The script to run the model is available [here](https://github.com/huggingface/diffusers/tree/main/examples/reinforcement_learning).

> [!TIP]
> Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-a-pipeline) section to learn how to efficiently load the same components into multiple pipelines.

## ValueGuidedRLPipeline[[diffusers.experimental.ValueGuidedRLPipeline]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.experimental.ValueGuidedRLPipeline</name><anchor>diffusers.experimental.ValueGuidedRLPipeline</anchor><source>https://github.com/huggingface/diffusers/blob/main/src/diffusers/experimental/rl/value_guided_sampling.py#L25</source><parameters>[{"name": "value_function", "val": ": UNet1DModel"}, {"name": "unet", "val": ": UNet1DModel"}, {"name": "scheduler", "val": ": DDPMScheduler"}, {"name": "env", "val": ""}]</parameters><paramsdesc>- **value_function** ([UNet1DModel](/docs/diffusers/main/en/api/models/unet#diffusers.UNet1DModel)) --
  A specialized UNet for fine-tuning trajectories base on reward.
- **unet** ([UNet1DModel](/docs/diffusers/main/en/api/models/unet#diffusers.UNet1DModel)) --
  UNet architecture to denoise the encoded trajectories.
- **scheduler** ([SchedulerMixin](/docs/diffusers/main/en/api/schedulers/overview#diffusers.SchedulerMixin)) --
  A scheduler to be used in combination with `unet` to denoise the encoded trajectories. Default for this
  application is [DDPMScheduler](/docs/diffusers/main/en/api/schedulers/ddpm#diffusers.DDPMScheduler).
- **env** () --
  An environment following the OpenAI gym API to act in. For now only Hopper has pretrained models.</paramsdesc><paramgroups>0</paramgroups></docstring>

Pipeline for value-guided sampling from a diffusion model trained to predict sequences of states.

This model inherits from [DiffusionPipeline](/docs/diffusers/main/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods
implemented for all pipelines (downloading, saving, running on a particular device, etc.).




</div>

<EditOnGithub source="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/value_guided_sampling.md" />