Without test classification and evaluation issues about kpconv-pytorch HOT 9 CLOSED

Hi @tangbohu,

I just added a piece of code in utils.tester.py, but it might have bugs, tell me if you are able to use it.

Best,
Hugues

from kpconv-pytorch.

Hi @HuguesTHOMAS ，
Thanks very much for your quick reply!

However, why you run the test for mutiple times as the code below:

while np.min(self.test_counts) < num_votes:
for batch in test_loader:
...

In my opinion, the results for the many different runs should be the same. Besides, I find it appears in the test code for all the tasks. Maybe I misunderstood your intention. Could you help me to explain it.

from kpconv-pytorch.

same of problem of results interpretation, i do not understand the result.

it would be nice if train_acc, test_acc for each epoch could be printed.

from kpconv-pytorch.

@tangbohu, Because of data augmentation, the result can be different every time you test a shape. THerefore I test all shapes more than once and average the probabilities. This is why the obtained accuracy is a "vote".

@shangguan91, The validation accuracy is printed once at the end of each epoch, when the validation is done. The full confusion matrix is stored in a text file in the log folder, and you can use the plot_convergence.py script to show a graph of the accuracy. As for the training accuracy, it is not a useful measure, so it is not fully computed. I only check the accuracy on each training batch to have an idea if the networks start converging.

from kpconv-pytorch.

@HuguesTHOMAS i understand.
so, how many times does it to take to train 50 epochs? it tooks like 15 minutes for first 5 epoch, i remember.

from kpconv-pytorch.

It depends on your computer. A few hours generally

from kpconv-pytorch.

@tangbohu, Because of data augmentation, the result can be different every time you test a shape. THerefore I test all shapes more than once and average the probabilities. This is why the obtained accuracy is a "vote".

Thanks Hugues for your kind reply!

But should we apply data augmentation during testing? In my previous work on 2D images, I apply data augmentation only for the training stage.

Besides, since with the following code in your ModelNet40Dataset

"self.epoch_n = min(self.num_models, config.validation_size * config.batch_num)", with setting config.validation_size = 200,
the evulation is only applied to a part of test set instead of all the test shapes.

from kpconv-pytorch.

@tangbohu
Did you solved this problem?

I'm confused too because of the evaluation accuracies are so different.
I also agree your opinion that the data augmentation should be applied during training and test should be performed on whole test set at once.

from kpconv-pytorch.

@tangbohu, sorry for the late reply, it is very common to apply data augmentation on test set. This vote strategy was first introduced by paper using image projection of the shapes (commonly referred as multiview networks as they use multiple views of a shape to classify it). Then people started to use same idea with 3D architectures. The "good practice" for such problems really depend on the applications. If you consider realtime applications like autonomous vehicles, then you cannot spare time to do augmentation on test set, but for offline applications like 3D mapping it is totally acceptable and even advised to ensure more consistency in the results

Concerning the part of the code you highlighted, this only concerns one epoch of validation, but the validation is applied at every epoch. We cannot spend to much time at every epoch to test all shapes, so we just test some of them, and at each epoch, I choose new part of the test set, so that every shape is tested eventually. In fact if you look at the ModelNet40 sampler, you will see that when self.use_potential is True, each shape of the test set is ensured to be tested the same number of times as the others.

@dogyoonlee, I hope that answer your question too.

from kpconv-pytorch.