Train, validation, test split about stpls3d HOT 18 CLOSED

Great, I'm glad to hear that, I don't have any open3d scripts for that. I usually write it out using https://github.com/meidachen/STPLS3D/blob/main/HAIS/mergeResult.py
and check it in CloudCompare.

from stpls3d.

Thank you for your interest in our work.

1. You are correct, this is where you can change the train/Val split.
2. Yes, but if the data you are testing is not from STPLS3D, you will need to make sure it has the same format as STPLS3D data.
3. Yes, because there is a data augmentation process in prepare_data_statistic_stpls3d.py, which has some randomness in it, and will produce slightly different statistics. It should be fine to reproduce our results.

from stpls3d.

Thanks for the fast reply!

3. It means that each time after running prepare_data_statistic_stpls3d.py updating statistics parameters is obligatory -- you made it sound rather optional in the instructions.

I'm running training now, will see the scores tomorrow :)

from stpls3d.

Thanks for the fast reply!

3. It means that each time after running prepare_data_statistic_stpls3d.py updating statistics parameters is obligatory -- you made it sound rather optional in the instructions.

I'm running training now, will see the scores tomorrow :)

Great! Looking forward to hearing your result! Just a heads up, it took me 2 days to train using the entire STPLS3D datasets on a 3090.
3) From my experiments, I tested the train/test process a few times by running prepare_data_statistic_stpls3d.py from scratch, the small differences in the statistic parameters didn't affect much on the results, so I made it optional.

from stpls3d.

Hi!
Trying to test with: CUDA_VISIBLE_DEVICES=1 python test.py --config config/hais_run_stpls3d.yaml --pretrain exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/hais_run_stpls3d-000000368.pth

gives out error:

python: src/hierarchical_aggregation/hierarchical_aggregation.cu:153: void hierarchical_aggregation_cuda(int, int, int*, int*, float*, int, int, int*, int*, float*, int*, int*): Assertion primary_num <= MAX_PRIMARY_NUM failed.

Logs seem more-or-less reasonable:

Do you know what is this error about? Does it mean primary predicted number of instances is larger than expected?

from stpls3d.

I haven't seen this error before, are you using all default settings from this repo? or are there any changes you made to it? I also saw this: hustvl/HAIS#21 , not sure if it is related but maybe worth a try.

from stpls3d.

Thank you, I found my mistake: I forgot to rebuild the hais_ops, now test works, and I'm retraining the model to full 500 epochs 😊

Do you have any open3d scripts to visualize segmentation results?

from stpls3d.

Thank you, I found my mistake: I forgot to rebuild the hais_ops, now test works, and I'm retraining the model to full 500 epochs 😊

Could you please also report your results here if possible?

Thank you!

from stpls3d.

The current results on 500 validation point clouds are the following:

  ################################################################
  what           :             AP         AP_50%         AP_25%
  ################################################################
  Building       :          0.666          0.716          0.726
  LowVegetation  :          0.290          0.452          0.554
  MediumVegetation:         0.256          0.379          0.439
  HighVegetation :          0.254          0.299          0.319
  Vehicle        :          0.782          0.880          0.911
  Truck          :          0.605          0.710          0.776
  Aircraft       :          0.370          0.548          0.667
  MilitaryVehicle:          0.320          0.420          0.531
  Bike           :          0.099          0.178          0.213
  Motorcycle     :          0.595          0.781          0.837
  LightPole      :          0.554          0.692          0.754
  StreetSgin     :          0.103          0.181          0.203
  Clutter        :          0.217          0.274          0.320
  Fence          :          0.324          0.532          0.704
  ----------------------------------------------------------------
 average        :          0.388          0.503          0.568

  semantic_segmantation_accuracy: 0.9427
  semantic_segmantation_mIoU: 0.4839

These are for the default statistics values - I returned to your values when had searched for the error source. Probably updating to recalculated values might give better results.

from stpls3d.

After updating statistics parameters the results have improved, the scores are a bit higher than ones reported in your paper, especially for some rare classes like Bike:

  ################################################################
  what           :             AP         AP_50%         AP_25%
  ################################################################
  Building       :          0.695          0.750          0.762
  LowVegetation  :          0.289          0.470          0.551
  MediumVegetation:         0.305         0.439           0.500
  HighVegetation :          0.251          0.295          0.319
  Vehicle        :          0.774          0.877          0.908
  Truck          :          0.550          0.648          0.713
  Aircraft       :          0.524          0.738          0.786
  MilitaryVehicle:          0.571          0.679          0.716
  Bike           :          0.166          0.298          0.356
  Motorcycle     :          0.566          0.743          0.800
  LightPole      :          0.537          0.686          0.738
  StreetSgin     :          0.142          0.237          0.275
  Clutter        :          0.267          0.312          0.363
  Fence          :          0.328          0.534          0.708
  ----------------------------------------------------------------
  average        :          0.426          0.550          0.607 
  
  semantic_segmantation_accuracy: 0.9414
  semantic_segmantation_mIoU: 0.4828

from stpls3d.

After updating statistics parameters the results have improved, the scores are a bit higher than ones reported in your paper, especially for some rare classes like Bike:

  ################################################################
  what           :             AP         AP_50%         AP_25%
  ################################################################
  Building       :          0.695          0.750          0.762
  LowVegetation  :          0.289          0.470          0.551
  MediumVegetation:         0.305         0.439           0.500
  HighVegetation :          0.251          0.295          0.319
  Vehicle        :          0.774          0.877          0.908
  Truck          :          0.550          0.648          0.713
  Aircraft       :          0.524          0.738          0.786
  MilitaryVehicle:          0.571          0.679          0.716
  Bike           :          0.166          0.298          0.356
  Motorcycle     :          0.566          0.743          0.800
  LightPole      :          0.537          0.686          0.738
  StreetSgin     :          0.142          0.237          0.275
  Clutter        :          0.267          0.312          0.363
  Fence          :          0.328          0.534          0.708
  ----------------------------------------------------------------
  average        :          0.426          0.550          0.607 
  
  semantic_segmantation_accuracy: 0.9414
  semantic_segmantation_mIoU: 0.4828

Great! Thanks for sharing!

from stpls3d.

Hi @Ritchizh,

I know you have been testing HAIS on this repo for STPLS3D and also your own datasets. We have recently identified a critical bug in our HAIS evaluation scripts. We have fixed the issue in this repo, but our previous benchmarks were incorrect. This bug doesn't affect training but only evaluation. I would like to leave a message here to you for your awareness. Please check our recent commit for the fix, and sorry for any inconvenience this maybe caused.
Below is the updated benchmark on STPLS3D:

from stpls3d.

Hi @meidachen! Thank you for the update, I would probably miss it otherwise. I will check the fixed version.

from stpls3d.

@Ritchizh @meidachen I'm glad we're doing the same research.I have the same problem，I want to ask you how to rebuild hais_ops? What command？When I rebuild hais_ops,just the 'build' file can't be overwritten，cause rebuild failure. Look forward to your reply！Thanks

from stpls3d.

Hi @Wangcongyu620!
To rebuild -

1. Go to lib/hais_ops folder, delete all files except for default ones which are in https://github.com/meidachen/STPLS3D/tree/main/HAIS/lib/hais_ops
2. Run commands as in the Setup environment instruction:

export CPLUS_INCLUDE_PATH={conda_env_path}/hais/include:$CPLUS_INCLUDE_PATH
python setup.py build_ext develop

Also related: hustvl/HAIS#30

from stpls3d.

OK, I tried it. It worked. Thanks. Can I understand that the larger the ‘MAX_PRIMARY_NUM’ parameter, the larger the area that can be processed？I want to know which parameters should be adjusted when adjusting this parameter? I'm afraid the effect will be worse if only this parameter is adjusted. Look forward to your reply

from stpls3d.

I'm afraid I can't tell for sure what is the best way of adjusting HAIS parameters.
The only place where ‘MAX_PRIMARY_NUM’ seems to be used is:

And the primary instances, if we look to the paper, are the initial instances that were clustered together from points. So my best guess is that by increasing ‘MAX_PRIMARY_NUM’ you allow the algorithm to predict more primary clusters.

from stpls3d.

I understand what you mean, I will try some more experiments. Thanks

from stpls3d.