Reproduce iCarl results about continual-learning-baselines HOT 10 CLOSED

This should be enough: https://github.com/ContinualAI/avalanche/blob/master/examples/icarl.py

from continual-learning-baselines.

Hi @qsunyuan , can you please open a new issue? @rudysemola maybe you can help answering?

from continual-learning-baselines.

I gladly try it

from continual-learning-baselines.

Hi! sorry to interrupt you.

I reproduced iCarl results (default args), but I cannot achieve the trarget results of 0.62 in this link, even I tried some different random seed.

I would like to know where did the value of avearge-acc 0.62 come from?

https://github.com/ContinualAI/reproducible-continual-learning/blob/eafcb218c08d6e6234799d0171b4a19bd6ac1d89/strategies/target_results.csv#L18

I also checked the original papaer. The last exp top1-test-acc is about 0.49 (amazing!)

I tried my own implementation code and the avalanche library. I can't reach such high results. Could u pls give some insights? I'm so confused right now.

Hope to get ur early replay. Thx.

from continual-learning-baselines.

Thx for ur quick replay. I will open a new issue.

from continual-learning-baselines.

Hi @qsunyuan.
If I remember well, the last result in the plot shown to you is the accuracy achieved for the last experience (0.49).
The metric used here is the average incremental accuracy defined by the authors in Section 4. (Experiments, Benchmark protocol part).
If you see the image, taken from the paper (Table 1 a), the result achieved in the paper achieved is 0.641 for 10 classes using this definition of the metric, not 0.49.
0.62 was achieved using the same definition in the paper.

from continual-learning-baselines.

If you want another similar code to reproduce the results in the paper (near to 0.641) you can use this old code, is basically the same but you never know maybe can work and help you ;)

https://github.com/rudysemola/reproducible-continual-learning/blob/024d26585dc9b3917d8827be376b57dd3b1eaa40/strategies/iCARL/experiment.py

from continual-learning-baselines.

Thx for ur help. It really helps me a lot. I will try the codes right now.
Thx again. Best wishes. Have a good day.

from continual-learning-baselines.

I tried ur link @rudysemola and this one https://github.com/ContinualAI/avalanche/blob/master/examples/icarl.py

I achieved a result of about 0.62.

Unfortunately I also did not achieve the result of 0.64.

from continual-learning-baselines.

Hi :) Minor changes in the performance with respect to the original paper are often due to slightly different training modalities (e.g. learning rate scheduler) which are not always easy to investigate or are not disclose in the paper. Therefore, as a policy, we allow for a 2% slack in accuracy during tests. Our target result for iCarl is 0.62. If you manage to close this gap, please let us know!

from continual-learning-baselines.