OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation
This is the implementation of ICRA 2021 paper "OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation" created by Yifan You*, Lin Shao*, Toki Migimatsu, and Jeannette Bohg.
Hanging objects is a common daily task. Our system helps robots learn to hang arbitrary objects onto a diverse set of supporting items such as racks and hooks. All hanging poses rendered here are outputs of our proposed pipeline on object-supporting item pairs unseen during training.
This repository provides data and code as follows.
data/ # contains all data. Details explained in Dataset Structure section
src/
scripts/ # contains code used to generate data
...
collect_pose_data_vary_scale.py # generates hanging poses
generate_takeoff_v2.py # checks if an object in the hanging pose can be taken off
generate_cp_acc_soft.py # generates soft contact points, given hanging poses
generate_partial_pc_soft.py # generates partial point clouds, given soft contact points
utils/ # something useful
lin_my/ # training/evaluation
runs/ # contains pretrained models, also where models/tensorboards/debugging info are saved during training/evaluation
pointnet4/ # code adapted from PointNet++ (https://github.com/charlesq34/pointnet2)
...
simple_dataset.py # simple dataset that loads supporting items, objects, and successful hanging poses
hang_dataset.py # dataset that loads supporting items, objects, successful hanging poses,
# contact points, and contact point correspondences
...
s1_train_matching.py # stage 1 training/evaluation
s2a_train.py # stage 2a training/evaluation
s2b_train_discrete.py # stage 2b training/evaluation
s3_rl_collect.py # stage 3 online data collection. also used for stage 3 evaluation
s3_rl_train.py # stage 3 online training
This code has been tested on Ubuntu 16.04 with Cuda 9.0, Python 3.6, and TensorFlow 1.12.
The dataset is organized as follows.
data/
...
geo_data/ # urdfs/meshes for objects and supporting items
geo_data_partial_cp_pad/ # partial point clouds for objects and supporting items
collection_result/ # successful hanging pose
collection_result_more/ # more successful hanging pose
collection_result_neg/ # unsuccessful hanging pose
collection_result_pene_big_neg_new # object poses w/o collision
collection_result_pene_big_pos_new # object poses w/ collision
dataset_cp/ # contact points and contact point correspondences for poses in collection_result/
dataset_cp_more/ # contact points and contact point correspondences for poses in collection_result_more/
This repo requires building PointNet++(https://github.com/charlesq34/pointnet2) in src/lin_my/pointnet4/. Please refer to PointNet++'s repo for building instructions.
To download the pretrained models, run
wget http://download.cs.stanford.edu/juno/omnihang/zipdir/runs.zip
unzip the downloaded runs.zip, and move runs/ to src/lin_my/runs.
We split the dataset into several zip files available for download. For all zip files downloaded, unzip the contents and move them under the data/ folder described in Dataset Structure section.
wget http://download.cs.stanford.edu/juno/omnihang/data/zipdir/geo_data.zip # contains the geo_data folder
wget http://download.cs.stanford.edu/juno/omnihang/data/zipdir/collection_result.zip # contains all data related to hanging poses (collection_result/ collection_result_more/, collection_result_neg/, collection_result_pene_big_neg_new/, collection_result_pene_big_pos_new/)
wget http://download.cs.stanford.edu/juno/omnihang/data/zipdir/dataset_cp.zip # contains all data related to contact points (dataset_cp, dataset_cp_more, and some other auxiliary files)
wget http://download.cs.stanford.edu/juno/omnihang/data/zipdir/geo_data_partial_cp_pad.zip # contains the geo_data_partial_cp_pad folder
Please post issues for questions and more helps on this Github repo page. We encourage using Github issues instead of sending us emails since your questions may benefit others.
@yifan-you-37 @linsats
@inproceedings{you2021omnihang,
title={OmniHang: Learning to Hang Arbitrary Objects Using Contact Point Correspondences and Neural Collision Estimation},
author={You, Yifan and Shao, Lin and Migimatsu, Toki and Bohg, Jeannette},
booktitle={2021 IEEE International Conference on Robotics and Automation (ICRA)},
year={2021},
organization={IEEE}}
MIT License
Please request in Github Issue for more code to release.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent