Giter Club home page Giter Club logo

cgf's Introduction

Continuous Grasping Function

This repository is the official implementation of the paper:

Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations
Jianglong Ye*, Jiashun Wang*, Binghao Huang, Yuzhe Qin, Xiaolong Wang
RA-L 2023, IROS 2023

Project Page / ArXiv / Video

Environment Setup

(Our code has been tested on Ubuntu 20.04, python 3.10, torch 1.13.1, CUDA 11.7 and RTX 3090)

To set up the environment, follow these steps:

conda create -n cgf python=3.10 -y && conda activate cgf
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia -y

pip install six numpy==1.23.1 tqdm pyyaml scipy opencv-python trimesh einops lxml transforms3d viser fvcore
pip install git+https://github.com/hassony2/chumpy.git
pip install git+https://github.com/hassony2/manopth
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py310_cu117_pyt1131/download.html
# the above command is for pytorch 1.13.1, CUDA 11.7, python 3.10, if you encounter any issues, please try install pytorch3d from source:
# pip install git+https://github.com/facebookresearch/pytorch3d.git

pip install git+https://github.com/UM-ARM-Lab/pytorch_kinematics

Data Preparation

We provide processed data for training CGF which can be downloaded from here and unzip it under PROJECT_ROOT/data/. The final file structure should look like this:

PROJECT_ROOT
├── ...
└── data
    ├── ...
    ├── geometry
    ├── mano_v1_2_models
    ├── original_root
    └── processed
        ├── ...
        ├── meta.json
        ├── pose_m_aug.npz
        └── pose_m_object_coord.npz

Alternatively, you can prepare the data by following the instructions below.

  1. Download DexYCB data from the official website to DEXYCB_DATA_ROOT and organize the data as follows:
    DEXYCB_DATA_ROOT
    ├── 20200709-subject-01/
    ├── 20200813-subject-02/
    ├── ...
    ├── calibration/
    └── models/
    
  2. Run the following script to process the data:
    python scripts/prepare_dexycb.py --data_root DEXYCB_DATA_ROOT
    python scripts/retargeting.py --mano_side left
    python scripts/retargeting.py --mano_side right
    The processed data will be saved to PROJECT_ROOT/data/processed. Retargeting results can be visualized by running the following command:
    python scripts/visualize_retargeting.py
    retargeting

Running

Training

To train the CGF model, run the following command:

# cd PROJECT_ROOT
python scripts/main.py

Sampling

To sample from the trained CGF model, run the following command (replace TIMESTAMP with the timestamp of the checkpoint in the PROJECT_ROOT/output directory):

We provide a pre-trained model which can be downloaded from here and unzip it under PROJECT_ROOT/output/.

# cd PROJECT_ROOT
python scripts/main.py --mode sample --ts TIMESTAMP

The sampled results will be saved to PROJECT_ROOT/output/TIMESTAMP/sample. They can be visualized by running the following command:

python scripts/visualize_sampling.py --ts TIMESTAMP

sampling

To evaluate the smoothness of the sampled grasps (table 1 in the paper), run the following command:

python scripts/eval_generation.py --ts TIMESTAMP

Note that the evaluation results may vary slightly due to the randomness in the sampling process.

Simulation

Our simulation environment is adapted from dex-hand-teleop. To setup the simulation environment, follow these steps:

pip install sapien==2.2.2 gym open3d

To run the simulation, run the following command (replace SAPIEN_SAMPLE_DATA_PATH with the path to a sample data in the PROJECT_ROOT/output/TIMESTAMP/sample/result_filter_sapien directory):

python scripts/sim.py --data_path SAPIEN_SAMPLE_DATA_PATH

Citation

@article{ye2023learning,
  title={Learning continuous grasping function with a dexterous hand from human demonstrations},
  author={Ye, Jianglong and Wang, Jiashun and Huang, Binghao and Qin, Yuzhe and Wang, Xiaolong},
  journal={IEEE Robotics and Automation Letters},
  year={2023}
}

cgf's People

Contributors

jianglongye avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.