Fork of lerobot/smolvla_base

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+collage_small.gif filter=lfs diff=lfs merge=lfs -text

Finetune_SmolVLA_notebook.ipynb

@@ -0,0 +1,214 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "NQUk3Y0WwYZ4"
+      },
+      "source": [
+        "# 🤗 x 🦾: Training SmolVLA with LeRobot Notebook\n",
+        "\n",
+        "Welcome to the **LeRobot SmolVLA training notebook**! This notebook provides a ready-to-run setup for training imitation learning policies using the [🤗 LeRobot](https://github.com/huggingface/lerobot) library.\n",
+        "\n",
+        "In this example, we train an `SmolVLA` policy using a dataset hosted on the [Hugging Face Hub](https://huggingface.co/), and optionally track training metrics with [Weights & Biases (wandb)](https://wandb.ai/).\n",
+        "\n",
+        "## ⚙️ Requirements\n",
+        "- A Hugging Face dataset repo ID containing your training data (`--dataset.repo_id=YOUR_USERNAME/YOUR_DATASET`)\n",
+        "- Optional: A [wandb](https://wandb.ai/) account if you want to enable training visualization\n",
+        "- Recommended: GPU runtime (e.g., NVIDIA A100) for faster training\n",
+        "\n",
+        "## ⏱️ Expected Training Time\n",
+        "Training with the `SmolVLA` policy for 20,000 steps typically takes **about 5 hours on an NVIDIA A100** GPU. On less powerful GPUs or CPUs, training may take significantly longer!\n",
+        "\n",
+        "## Example Output\n",
+        "Model checkpoints, logs, and training plots will be saved to the specified `--output_dir`. If `wandb` is enabled, progress will also be visualized in your wandb project dashboard.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MOJyX0CnwA5m"
+      },
+      "source": [
+        "## Install conda\n",
+        "This cell uses `condacolab` to bootstrap a full Conda environment inside Google Colab.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QlKjL1X5t_zM"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install -q condacolab\n",
+        "import condacolab\n",
+        "condacolab.install()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DxCc3CARwUjN"
+      },
+      "source": [
+        "## Install LeRobot\n",
+        "This cell clones the `lerobot` repository from Hugging Face, installs FFmpeg (version 7.1.1), and installs the package in editable mode.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dgLu7QT5tUik"
+      },
+      "outputs": [],
+      "source": [
+        "!git clone https://github.com/huggingface/lerobot.git\n",
+        "!conda install ffmpeg=7.1.1 -c conda-forge\n",
+        "!cd lerobot && pip install -e ."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Q8Sn2wG4wldo"
+      },
+      "source": [
+        "## Weights & Biases login\n",
+        "This cell logs you into Weights & Biases (wandb) to enable experiment tracking and logging."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PolVM_movEvp"
+      },
+      "outputs": [],
+      "source": [
+        "!wandb login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zTWQAgX9xseE"
+      },
+      "source": [
+        "## Install SmolVLA dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DiHs0BKwxseE"
+      },
+      "outputs": [],
+      "source": [
+        "!cd lerobot && pip install -e \".[smolvla]\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "IkzTo4mNwxaC"
+      },
+      "source": [
+        "## Start training SmolVLA with LeRobot\n",
+        "\n",
+        "This cell runs the `train.py` script from the `lerobot` library to train a robot control policy.  \n",
+        "\n",
+        "Make sure to adjust the following arguments to your setup:\n",
+        "\n",
+        "1. `--dataset.repo_id=YOUR_HF_USERNAME/YOUR_DATASET`:  \n",
+        "   Replace this with the Hugging Face Hub repo ID where your dataset is stored, e.g., `pepijn223/il_gym0`.\n",
+        "\n",
+        "2. `--batch_size=64`: means the model processes 64 training samples in parallel before doing one gradient update. Reduce this number if you have a GPU with low memory.\n",
+        "\n",
+        "3. `--output_dir=outputs/train/...`:  \n",
+        "   Directory where training logs and model checkpoints will be saved.\n",
+        "\n",
+        "4. `--job_name=...`:  \n",
+        "   A name for this training job, used for logging and Weights & Biases.\n",
+        "\n",
+        "5. `--policy.device=cuda`:  \n",
+        "   Use `cuda` if training on an NVIDIA GPU. Use `mps` for Apple Silicon, or `cpu` if no GPU is available.\n",
+        "\n",
+        "6. `--wandb.enable=true`:  \n",
+        "   Enables Weights & Biases for visualizing training progress. You must be logged in via `wandb login` before running this."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZO52lcQtxseE"
+      },
+      "outputs": [],
+      "source": [
+        "!cd lerobot && python lerobot/scripts/train.py \\\n",
+        "  --policy.path=lerobot/smolvla_base \\\n",
+        "  --dataset.repo_id=${HF_USER}/mydataset \\\n",
+        "  --batch_size=64 \\\n",
+        "  --steps=20000 \\\n",
+        "  --output_dir=outputs/train/my_smolvla \\\n",
+        "  --job_name=my_smolvla_training \\\n",
+        "  --policy.device=cuda \\\n",
+        "  --wandb.enable=true"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2PBu7izpxseF"
+      },
+      "source": [
+        "## Login into Hugging Face Hub\n",
+        "Now after training is done login into the Hugging Face hub and upload the last checkpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8yu5khQGIHi6"
+      },
+      "outputs": [],
+      "source": [
+        "!huggingface-cli login"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zFMLGuVkH7UN"
+      },
+      "outputs": [],
+      "source": [
+        "!huggingface-cli upload ${HF_USER}/my_smolvla \\\n",
+        "  /content/lerobot/outputs/train/my_smolvla/checkpoints/last/pretrained_model"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "gpuType": "A100",
+      "machine_shape": "hm",
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

README.md

@@ -0,0 +1,60 @@
+---
+pipeline_tag: robotics
+tags:
+- smolvla
+library_name: lerobot
+datasets:
+- lerobot/svla_so101_pickplace
+---
+
+## SmolVLA: A vision-language-action model for affordable and efficient robotics
+
+Resources and technical documentation:
+
+[SmolVLA Paper](https://huggingface.co/papers/2506.01844)
+
+[SmolVLA Blogpost](https://huggingface.co/blog/smolvla)
+
+[Code](https://github.com/huggingface/lerobot/blob/main/lerobot/common/policies/smolvla/modeling_smolvla.py)
+
+[Train using Google Colab Notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/lerobot/training-smolvla.ipynb#scrollTo=ZO52lcQtxseE)
+
+[SmolVLA HF Documentation](https://huggingface.co/docs/lerobot/smolvla)
+
+Designed by Hugging Face.
+
+This model has 450M parameters in total.
+You can use inside the [LeRobot library](https://github.com/huggingface/lerobot).
+
+Before proceeding to the next steps, you need to properly install the environment by following [Installation Guide](https://huggingface.co/docs/lerobot/installation) on the docs.
+
+Install smolvla extra dependencies:
+```bash
+pip install -e ".[smolvla]"
+```
+
+Example of finetuning the smolvla pretrained model (`smolvla_base`):
+```bash
+python lerobot/scripts/train.py \
+  --policy.path=lerobot/smolvla_base \
+  --dataset.repo_id=lerobot/svla_so101_pickplace \
+  --batch_size=64 \
+  --steps=20000 \
+  --output_dir=outputs/train/my_smolvla \
+  --job_name=my_smolvla_training \
+  --policy.device=cuda \
+  --wandb.enable=true
+```
+
+Example of finetuning the smolvla neural network with pretrained VLM and action expert
+intialized from scratch:
+```bash
+python lerobot/scripts/train.py \
+  --dataset.repo_id=lerobot/svla_so101_pickplace \
+  --batch_size=64 \
+  --steps=200000 \
+  --output_dir=outputs/train/my_smolvla \
+  --job_name=my_smolvla_training \
+  --policy.device=cuda \
+  --wandb.enable=true
+```

config.json

@@ -0,0 +1,96 @@
+{
+    "type": "smolvla",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.camera1": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        },
+        "observation.images.camera2": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        },
+        "observation.images.camera3": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "push_to_hub": true,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "chunk_size": 50,
+    "n_action_steps": 50,
+    "normalization_mapping": {
+        "VISUAL": "IDENTITY",
+        "STATE": "MEAN_STD",
+        "ACTION": "MEAN_STD"
+    },
+    "max_state_dim": 32,
+    "max_action_dim": 32,
+    "resize_imgs_with_padding": [
+        512,
+        512
+    ],
+    "empty_cameras": 0,
+    "adapt_to_pi_aloha": false,
+    "use_delta_joint_actions_aloha": false,
+    "tokenizer_max_length": 48,
+    "num_steps": 10,
+    "use_cache": true,
+    "freeze_vision_encoder": true,
+    "train_expert_only": true,
+    "train_state_proj": true,
+    "optimizer_lr": 0.0001,
+    "optimizer_betas": [
+        0.9,
+        0.95
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 1e-10,
+    "optimizer_grad_clip_norm": 10,
+    "scheduler_warmup_steps": 1000,
+    "scheduler_decay_steps": 30000,
+    "scheduler_decay_lr": 2.5e-06,
+    "vlm_model_name": "HuggingFaceTB/SmolVLM2-500M-Video-Instruct",
+    "load_vlm_weights": true,
+    "add_image_special_tokens": false,
+    "attention_mode": "cross_attn",
+    "prefix_length": 0,
+    "pad_language_to": "max_length",
+    "num_expert_layers": 0,
+    "num_vlm_layers": 16,
+    "self_attn_every_n_layers": 2,
+    "expert_width_multiplier": 0.75,
+    "min_period": 0.004,
+    "max_period": 4.0
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:979445bcff442b0e95e5187dd947db3324a14d737477fa4828323337d27e25df
+size 906712520

policy_postprocessor.json

@@ -0,0 +1,32 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:490ab239d96e263687c0b2e386a0afbc235a2eceb9857c36ed32f2f162a3e7c8
+size 640

policy_preprocessor.json

@@ -0,0 +1,87 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "smolvla_new_line_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "tokenizer_processor",
+      "config": {
+        "max_length": 48,
+        "task_key": "task",
+        "padding_side": "right",
+        "padding": "max_length",
+        "truncation": true,
+        "tokenizer_name": "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
+      }
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              6
+            ]
+          },
+          "observation.image2": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "observation.image": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "observation.image3": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_preprocessor_step_5_normalizer_processor.safetensors"
+    }
+  ]
+}

policy_preprocessor_step_5_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:490ab239d96e263687c0b2e386a0afbc235a2eceb9857c36ed32f2f162a3e7c8
+size 640