MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving
⭐ Our MWSIS is acceped by the AAAI24.
-
Release the MWSIS-pts code.
-
Improve data processing
-
Reorganize the current code (pts).
-
Replace the CCL.
-
Release the MWSIS-img code which is same as the MWSIS-pts.
- [24-01-10] The code for MWSIS-img is released here.
- [23-12-16] The code for MWSIS-pts is released here.
# python version is 3.7
# install torch
pip3 install torch==1.8.2 torchvision==0.9.2 torchaudio==0.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cu111
# install mmdet3d==1.0.0rc5 dependency
pip install mmcv-full==1.6.0 mmsegmentation==0.27.0 mmdet==2.25.1 spconv-cu111 open3d minio==7.1.11 pymongo==3.12.3 waymo-open-dataset-tf-2-6-0==1.4.9
# install mmdet3d
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v1.0.0rc5
pip install -e .
# clone the mwsis repo
git clone https://github.com/jiangxb98/mwsis-plugin.git
cd mwsis-plugin
# install pairwise_loss
cd boxinst-plugin-main
pip install -e .
mv boxinst_plugin/ops/ /mwsis_plugin/ops
# install torchex https://github.com/tusen-ai/SST
cd torchex
pip install -e .
# install torch scatter
pip install torch-scatter==2.1.0
# install easydict
pip install easydict
# download the resnet50 checkpoint
wget https://download.openmmlab.com/pretrain/third_party/resnet50_msra-5891d200.pth
mv .pth .cache/torch/hub/checkpoints/Train data
Please follow the tutorial given by mmdet3d to process the data, and then use our waymo conversion code to process the data. (Note: You should mv ./mwsis_plugin/tools/data_converter/waymo_converter_local.py ./tools/data_converter)
Val data generation
Please use mwsis_plugin/tools/data_converter/parallel_mask3d_eval_local.py
Note:
-
About CCL
Please use funcition
connected_components()to replace functionvoxel_spccl3d(). Later, we will change the CCL operator to the operator in the torchex library. Currently, sicpy library is used instead. -
Warning
Through experiment, we found that the torchex.connected_components unable to get the same results as the scipy.sparse.csgraph.connected_components.
# signle gpu
python tools/train.py mwsis/config/mwsis_pts_model_config.py --work-dir <your filepath>
# multi gpu
bash tools/dist_train.sh ...This project is based on the following codebases.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent