Open source license?

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Created by Ziyi Wang*, Xumin Yu*, Yongming Rao*, Jie Zhou, Jiwen Lu

This repository is an official implementation of P2P (NeurIPS 2022 Spotlight).

P2P is a framework to leverage pre-trained image models for 3D analysis. We transform point clouds into colorful images with newly proposed geometry-preserved projection and geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud understanding tasks.

[arXiv][Project Page][Models]

Preparation

Installation Prerequisites

Python 3.9
CUDA 11.3
PyTorch 1.11.1
timm 0.5.4
torch_scatter
pointnet2_ops
cv2, sklearn, yaml, h5py

conda create -n p2p python=3.9
conda activate p2p
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3

mkdir lib
cd lib
git clone https://github.com/erikwijmans/Pointnet2_PyTorch.git
cd Pointnet2_PyTorch
pip install pointnet2_ops_lib/.
cd ../..

pip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu113.html
pip install timm==0.5.4
pip install opencv-python
pip install scikit-learn, h5py
conda install pyyaml, tqdm

Data Preparation

Download the processed ModelNet40 dataset from [Google Drive][Tsinghua Cloud][BaiDuYun](code:4u1e). Or you can download the offical ModelNet from here, and process it by yourself.
Download the official ScanObjectNN dataset from here.

The data is expected to be in the following file structure:

P2P/
|-- config/
|-- data/
    |-- ModelNet40/
        |-- modelnet40_shape_names.txt
        |-- modelnet_train.txt
        |-- modelnet_test.txt
        |-- modelnet40_train_8192pts_fps.dat
        |-- modelnet40_test_8192pts_fps.dat
    |-- ScanObjectNN/
        |-- main_split/
            |-- training_objectdataset_augmentedrot_scale75.h5
            |-- test_objectdataset_augmentedrot_scale75.h5
|-- dataset/

Usage

Train

bash tool/train.sh EXP_NAME CONFIG_PATH DATASET

For example, to train P2P model with ConvNeXt-B-1k as base model on the ModelNet40 dataset:

bash tool/train.sh p2p_ConvNeXt-B-1k config/ModelNet40/p2p_ConvNeXt-B-1k.yaml ModelNet40

Test

bash tool/test.sh EXP_NAME CONFIG_PATH DATASET

For example, to test P2P model with ConvNeXt-B-1k as base model the ModelNet40 dataset:

bash tool/test.sh p2p_ConvNeXt-B-1k config/ModelNet40/p2p_ConvNeXt-B-1k.yaml ModelNet40

Reproduce

bash tool/reproduce.sh DATASET MODEL

For example, to reproduce results of P2P model with ConvNeXt-B-1k as base model on the ModelNet40 dataset with our provided checkpoint:

bash tool/reproduce.sh ModelNet40 ConvNeXt-B-1k

Results

Quantitative Results

We provide pretrained P2P models:

Image Model	ModelNet Acc.	ScanObjectNN Acc.
HorNet-L-22k-mlp	94.0 [Google / Tsinghua Cloud]	89.3 [Google / Tsinghua Cloud]
ResNet-18	91.6 [Google / Tsinghua Cloud]	82.6 [Google / Tsinghua Cloud]
ResNet-50	92.5 [Google / Tsinghua Cloud]	85.8 [Google / Tsinghua Cloud]
ResNet-101	93.1 [Google / Tsinghua Cloud]	87.4 [Google / Tsinghua Cloud]
ConvNeXt-T-1k	92.6 [Google / Tsinghua Cloud]	84.9 [Google / Tsinghua Cloud]
ConvNeXt-S-1k	92.8 [Google / Tsinghua Cloud]	85.3 [Google / Tsinghua Cloud]
ConvNeXt-B-1k	93.0 [Google / Tsinghua Cloud]	85.7 [Google / Tsinghua Cloud]
ConvNeXt-L-1k	93.2 [Google / Tsinghua Cloud]	86.2 [Google / Tsinghua Cloud]
ViT-T-1k	91.5 [Google / Tsinghua Cloud]	79.7 [Google / Tsinghua Cloud]
ViT-S-1k	91.8 [Google / Tsinghua Cloud]	81.6 [Google / Tsinghua Cloud]
ViT-B-1k	92.7 [Google / Tsinghua Cloud]	83.4 [Google / Tsinghua Cloud]
Swin-T-1k	92.1 [Google / Tsinghua Cloud]	82.9 [Google / Tsinghua Cloud]
Swin-S-1k	92.5 [Google / Tsinghua Cloud]	83.8 [Google / Tsinghua Cloud]
Swin-B-1k	92.6 [Google / Tsinghua Cloud]	84.6 [Google / Tsinghua Cloud]

To train our P2P framework with HorNet-L-22k as the image model, please download the pre-trained weight from the official HorNet repo. Please organize the directory as the following structure:

    P2P/
    |-- pretrained/
        |-- hornet_large_gf_in22k.pth
        |-- reproduce/
            |-- ckpt/
                |-- ModelNet40/
                    |-- ConvNeXt-B-1k-ModelNet40.pth
                |-- ScanObjectNN/
                    |-- ConvNeXt-B-1k-ScanObjectNN.pth

Visualization Results

Citation

If you find our work useful in your research, please consider citing:

@article{wang2022p2p,
title={P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting},
author={Wang, Ziyi and Yu, Xumin and Rao, Yongming and Zhou, Jie and Lu, Jiwen},
journal={arXiv preprint arXiv:2208.02812},
year={2022}
}

Acknowledgements

Our code is inspired by BPNet, Point-BERT.

wangzy22 / p2p Goto Github PK

p2p's Introduction

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Preparation

Installation Prerequisites

Data Preparation

Usage

Train

Test

Reproduce

Results

Quantitative Results

Visualization Results

Citation

Acknowledgements

p2p's People

Contributors

Stargazers

Watchers

Forkers

p2p's Issues

Recommend Projects

Recommend Topics

Recommend Org