ICML 2024
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan
Tabel of Contents
- [2024/06] Training and inference code for goal generation diffusion models are released.
- [2024/05] 3D-VLA is accepted to ICML 2024!
- [2024/03] Paper is on arXiv.
3D-VLA is a framework that connects vision-language-action (VLA) models to the 3D physical world. Unlike traditional 2D models, 3D-VLA integrates 3D perception, reasoning, and action through a generative world model, similar to human cognitive processes. It is built on the 3D-LLM and uses interaction tokens to engage with the environment. Embodied diffusion models are trained and aligned with the LLM to predict goal images and point clouds.
conda create -n 3dvla python=3.9
conda activate 3dvla
pip install -r requirements.txt
We will update the file structure and the installation process in the future.
-
Train the goal image latent diffusion model with the following command:
bash launcher/train_ldm.sh [NUM_GPUS] [NUM_NODES]
If you want to include depth information, you could add
--include_depth
to the command in thetrain_ldm.sh
file. -
Then you could generate the goal images:
python inference_ldm_goal_image.py --ckpt_folder lavis/output/LDM/pix2pix/runs (--include_depth)
The results will be saved in the
lavis/output/LDM/pix2pix/results
folder.
-
Train the goal point cloud diffusion model (finetuning the pretrained Point-E model):
bash launcher/train_pe.sh [NUM_GPUS] [NUM_NODES]
We will soon support the FSDP training for the goal point cloud generation.
-
Inferece the goal point cloud with the following command:
python inference_pe_goal_pcd.py
If you want to use multiple GPUs, use
torchrun --nproc_per_node=[NUM_GPUS] --master_port=[PORT] inference_pe_goal_pcd.py
instead.
- Train our 3D-VLA model:
bash launcher/train_llm.sh [NUM_GPUS] [NUM_NODES]
@article{zhen20243dvla,
author = {Zhen, Haoyu and Qiu, Xiaowen and Chen, Peihao and Yang, Jincheng and Yan, Xin and Du, Yilun and Hong, Yining and Gan, Chuang},
title = {3D-VLA: 3D Vision-Language-Action Generative World Model},
journal = {arXiv preprint arXiv:2403.09631},
year = {2024},
}
Here we would like to thank the following resources for their great work:
- SAM, ConceptFusion and 3D-CLR for Data Processing.
- Diffusers, InstructPix2Pix, StableDiffusion and Point-E for the Diffusion Model.
- LAVIS and 3D-LLM for the Codebase and Architecture.
- OpenX for Dataset.
- RLBench and Hiveformer for Evaluation.