#
SmolVLA

SmolVLA is Hugging Face’s lightweight foundation model for robotics. Designed for easy fine-tuning on LeRobot datasets, it helps accelerate your development!

Figure 1. SmolVLA takes as input (i) multiple cameras views, (ii) the robot’s current sensorimotor state, and (iii) a natural language instruction, encoded into contextual features used to condition the action expert when generating an action chunk.

##
Set Up Your Environment

1. Install LeRobot by following our Installation Guide.

2. Install SmolVLA dependencies by running:

       pip install -e ".[smolvla]"

##
Collect a dataset

SmolVLA is a base model, so fine-tuning on your own data is required for optimal performance in your setup. We recommend recording ~50 episodes of your task as a starting point. Follow our guide to get started: Recording a Dataset

> In your dataset, make sure to have enough demonstrations per each variation (e.g. the cube position on the table if it is cube pick-place task) you are introducing.

> We recommend checking out the dataset linked below for reference that was used in the SmolVLA paper:

> 🔗 SVLA SO100 PickPlace

> In this dataset, we recorded 50 episodes across 5 distinct cube positions. For each position, we collected 10 episodes of pick-and-place interactions. This structure, repeating each variation several times, helped the model generalize better. We tried similar dataset with 25 episodes, and it was not enough leading to a bad performance. So, the data quality and quantity is definitely a key. After you have your dataset available on the Hub, you are good to go to use our finetuning script to adapt SmolVLA to your application.

##
Finetune SmolVLA on your data

Use `smolvla_base`, our pretrained 450M model, and fine-tune it on your data. Training the model for 20k steps will roughly take ~4 hrs on a single A100 GPU. You should tune the number of steps based on performance and your use-case.

If you don’t have a gpu device, you can train using our notebook on

Pass your dataset to the training script using `--dataset.repo_id`. If you want to test your installation, run the following command where we use one of the datasets we collected for the SmolVLA Paper.

    cd lerobot && lerobot-train \
      --policy.path=lerobot/smolvla_base \
      --dataset.repo_id=${HF_USER}/mydataset \
      --batch_size=64 \
      --steps=20000 \
      --output_dir=outputs/train/my_smolvla \
      --job_name=my_smolvla_training \
      --policy.device=cuda \
      --wandb.enable=true

> You can start with a small batch size and increase it incrementally, if the GPU allows it, as long as loading times remain short.

Fine-tuning is an art. For a complete overview of the options for finetuning, run

    lerobot-train --help

Figure 2: Comparison of SmolVLA across task variations. From left to right: (1) pick-place cube counting, (2) pick-place cube counting, (3) pick-place cube counting under perturbations, and (4) generalization on pick-and-place of the lego block with real-world SO101.

##
Evaluate the finetuned model and run it in real-time

Similarly for when recording an episode, it is recommended that you are logged in to the HuggingFace Hub. You can follow the corresponding steps: Record a dataset. Once you are logged in, you can run inference in your setup by doing:

    lerobot-record \
      --robot.type=so101_follower \
      --robot.port=/dev/ttyACM0 \ # <- Use your port
      --robot.id=my_blue_follower_arm \ # <- Use your robot id
      --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \ # <- Use your cameras
      --dataset.single_task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording
      --dataset.repo_id=${HF_USER}/eval_DATASET_NAME_test \  # <- This will be the dataset name on HF Hub
      --dataset.episode_time_s=50 \
      --dataset.num_episodes=10 \
      # <- Teleop optional if you want to teleoperate in between episodes \
      # --teleop.type=so100_leader \
      # --teleop.port=/dev/ttyACM0 \
      # --teleop.id=my_red_leader_arm \
      --policy.path=HF_USER/FINETUNE_MODEL_NAME # <- Use your fine-tuned model

Depending on your evaluation setup, you can configure the duration and the number of episodes to record for your evaluation suite.

Update on GitHub