Continual learning baselines and strategies from popular papers, using Avalanche. We include EWC, SI, GEM, AGEM, LwF, iCarl, GDumb, and other strategies.
Home Page: https://avalanche.continualai.org/
License: MIT License
Currently, we still face a performance gap for some of the existing strategies.
The expected performance can be found in the comments of the related experiments folder.
Any help in closing the gap is welcome. Just comment this issue and I will assign you to that strategy.
List of strategies to "fix":
The target results depend on the generator that is being used (as well as other factors, such as how much replay data is generated and whether the replay data is class-balanced or not).
This paper lists the various results for different generative models.
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
Test
Some tests failed. Check server log for details.
Traceback (most recent call last):
File "/home/acossu/continual-learning-baselines/tests/lwf/experiment.py", line 33, in test_smnist
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.31 != 0.1944 within 0.03 delta (0.11560000000000001 difference)
Ran 1 test in 124.929s
FAILED (failures=1)
issue test
Hi, I am running some experiments to compare synaptic intelligence and naive fine-tuning on different benchmarks, including SplitMNIST, PermutedMNIST, and a custom dataset benchmark of non-iid datasets. I observed that the performance of synaptic intelligence mirrors exactly the performance of the naive fine-tuning strategy. Is this performance expected?
We are missing baselines for Replay with Reservoir Sampling and Class-Balanced Reservoir Sampling.
I noticed something strange in my own experiments, related to a change I made a while ago to LwF in the avalanche master branch. Basically, right now the distillation is applied only to the previously active units. Formally, this is the closest solution to the original paper (which only uses multiheads). However, distilling on all the units (as we did previously) results in a better accuracy, probably because of the additional penalization to the new units.
The details are as follows.
In experiments.slipt_mnist.naive module, the 'task_incremental': False in line 18 is not corresponding to the return_task_id=args.task_incremental in line 24, which will cause AttributeError: 'types.SimpleNamespace' object has no attribute 'task_incremental' on my computer. I think it's a typo in line 18, but it has been here for a quite long time.
I am using Python 3.9, and I suppose it can be fixed in one second, literally.
Some tests failed. Check server log for details.
This requires further investigation
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
I think they are task-incremental, since AGEM ones are task-incremental and similar, and they look quite high for task-incremental
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.agem.AGEMPlugin object at 0x7f52f7dbe700> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.AGEM object at 0x7f52f7dbe6d0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.agem.AGEMPlugin object at 0x7f52f77ff3a0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.AGEM object at 0x7f52f7803610>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.cope.CoPEPlugin object at 0x7f531d1eb3a0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.Naive object at 0x7f531d1eb640>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/evaluation.py:85: UserWarning: No benchmark provided to the evaluation plugin. Metrics may be computed on inconsistent portion of streams, use at your own risk.
warnings.warn(
/home/acossu/reproducible-continual-learning/strategies/dslda/experiment.py:59: UserWarning: The Deep SLDA example is not perfectly aligned with the paper implementation since it does not use a base initialization phase and instead starts streming from pre-trained weights. Performance should still match.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.ewc.EWCPlugin object at 0x7f5367f4e910> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.EWC object at 0x7f5367f4e610>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gdumb.GDumbPlugin object at 0x7f5367b414f0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.GDumb object at 0x7f5367b41d60>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gem.GEMPlugin object at 0x7f5367af79a0> implements incompatible callbacks for template <strategies.gem.experiment.GEM_reduced object at 0x7f5367af7c10>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gem.GEMPlugin object at 0x7f53675ab4c0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.GEM object at 0x7f53675ab8b0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/evaluation.py:85: UserWarning: No benchmark provided to the evaluation plugin. Metrics may be computed on inconsistent portion of streams, use at your own risk.
warnings.warn(
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gss_greedy.GSS_greedyPlugin object at 0x7f5367c33c10> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.GSS_greedy object at 0x7f5367c33c40>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lwf.LwFPlugin object at 0x7f5367d1c6d0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.LwF object at 0x7f5367d1c280>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lwf.LwFPlugin object at 0x7f5367a539a0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.LwF object at 0x7f5367a53220>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lwf.LwFPlugin object at 0x7f5367b41040> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.LwF object at 0x7f5368474250>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.mas.MASPlugin object at 0x7f536760a820> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.MAS object at 0x7f536760a3a0>. This may result in errors.
warnings.warn(
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.09it/s]
8%|▊ | 4/50 [00:00<00:04, 9.99it/s]
10%|█ | 5/50 [00:00<00:04, 9.94it/s]
12%|█▏ | 6/50 [00:00<00:04, 9.60it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.01it/s]
20%|██ | 10/50 [00:00<00:03, 10.20it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.35it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.48it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.55it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.58it/s]
40%|████ | 20/50 [00:01<00:02, 10.59it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.50it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.55it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.59it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.61it/s]
60%|██████ | 30/50 [00:02<00:01, 10.66it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.71it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.73it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.71it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.65it/s]
80%|████████ | 40/50 [00:03<00:00, 10.66it/s]
84%|████████▍ | 42/50 [00:03<00:00, 10.67it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.70it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.70it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.61it/s]
100%|██████████| 50/50 [00:04<00:00, 10.62it/s]
100%|██████████| 50/50 [00:04<00:00, 10.53it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.33it/s]
8%|▊ | 4/50 [00:00<00:04, 10.32it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.29it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.26it/s]
20%|██ | 10/50 [00:00<00:03, 10.24it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.29it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.32it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.32it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.27it/s]
40%|████ | 20/50 [00:01<00:02, 10.28it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.27it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.31it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.31it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.28it/s]
60%|██████ | 30/50 [00:02<00:01, 10.32it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.29it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.28it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.29it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.29it/s]
80%|████████ | 40/50 [00:03<00:00, 10.31it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.34it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.39it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.39it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.38it/s]
100%|██████████| 50/50 [00:04<00:00, 10.38it/s]
100%|██████████| 50/50 [00:04<00:00, 10.32it/s]
0%| | 0/50 [00:00<?, ?it/s]
2%|▏ | 1/50 [00:00<00:04, 9.91it/s]
6%|▌ | 3/50 [00:00<00:04, 10.26it/s]
10%|█ | 5/50 [00:00<00:04, 10.37it/s]
14%|█▍ | 7/50 [00:00<00:04, 10.39it/s]
18%|█▊ | 9/50 [00:00<00:03, 10.40it/s]
22%|██▏ | 11/50 [00:01<00:03, 10.43it/s]
26%|██▌ | 13/50 [00:01<00:03, 10.47it/s]
30%|███ | 15/50 [00:01<00:03, 10.45it/s]
34%|███▍ | 17/50 [00:01<00:03, 10.46it/s]
38%|███▊ | 19/50 [00:01<00:02, 10.47it/s]
42%|████▏ | 21/50 [00:02<00:02, 10.44it/s]
46%|████▌ | 23/50 [00:02<00:02, 10.44it/s]
50%|█████ | 25/50 [00:02<00:02, 10.48it/s]
54%|█████▍ | 27/50 [00:02<00:02, 10.49it/s]
58%|█████▊ | 29/50 [00:02<00:02, 10.45it/s]
62%|██████▏ | 31/50 [00:02<00:01, 10.45it/s]
66%|██████▌ | 33/50 [00:03<00:01, 10.42it/s]
70%|███████ | 35/50 [00:03<00:01, 10.36it/s]
74%|███████▍ | 37/50 [00:03<00:01, 10.38it/s]
78%|███████▊ | 39/50 [00:03<00:01, 10.38it/s]
82%|████████▏ | 41/50 [00:03<00:00, 10.40it/s]
86%|████████▌ | 43/50 [00:04<00:00, 10.40it/s]
90%|█████████ | 45/50 [00:04<00:00, 10.39it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.40it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.38it/s]
100%|██████████| 50/50 [00:04<00:00, 10.41it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.26it/s]
8%|▊ | 4/50 [00:00<00:04, 10.26it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.31it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.37it/s]
20%|██ | 10/50 [00:00<00:03, 10.35it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.38it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.38it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.40it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.42it/s]
40%|████ | 20/50 [00:01<00:02, 10.40it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.40it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.40it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.37it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.35it/s]
60%|██████ | 30/50 [00:02<00:01, 10.36it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.33it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.32it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.28it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.26it/s]
80%|████████ | 40/50 [00:03<00:00, 10.24it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.26it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.24it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.30it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.33it/s]
100%|██████████| 50/50 [00:04<00:00, 10.34it/s]
100%|██████████| 50/50 [00:04<00:00, 10.34it/s]
0%| | 0/50 [00:00<?, ?it/s]
2%|▏ | 1/50 [00:00<00:05, 9.55it/s]
4%|▍ | 2/50 [00:00<00:05, 9.51it/s]
6%|▌ | 3/50 [00:00<00:04, 9.49it/s]
8%|▊ | 4/50 [00:00<00:04, 9.44it/s]
10%|█ | 5/50 [00:00<00:04, 9.45it/s]
12%|█▏ | 6/50 [00:00<00:04, 9.38it/s]
14%|█▍ | 7/50 [00:00<00:04, 9.41it/s]
16%|█▌ | 8/50 [00:00<00:04, 9.37it/s]
18%|█▊ | 9/50 [00:00<00:04, 9.40it/s]
20%|██ | 10/50 [00:01<00:04, 9.36it/s]
22%|██▏ | 11/50 [00:01<00:04, 9.40it/s]
24%|██▍ | 12/50 [00:01<00:04, 9.24it/s]
26%|██▌ | 13/50 [00:01<00:04, 9.24it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.22it/s]
30%|███ | 15/50 [00:01<00:03, 9.33it/s]
32%|███▏ | 16/50 [00:01<00:03, 9.38it/s]
34%|███▍ | 17/50 [00:01<00:03, 9.26it/s]
36%|███▌ | 18/50 [00:01<00:03, 9.35it/s]
40%|████ | 20/50 [00:02<00:03, 9.76it/s]
44%|████▍ | 22/50 [00:02<00:02, 9.96it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.09it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.10it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.19it/s]
60%|██████ | 30/50 [00:03<00:01, 10.27it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.35it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.38it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.39it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.39it/s]
80%|████████ | 40/50 [00:04<00:00, 10.42it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.42it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.41it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.42it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.46it/s]
100%|██████████| 50/50 [00:05<00:00, 10.46it/s]
100%|██████████| 50/50 [00:05<00:00, 10.00it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.14it/s]
8%|▊ | 4/50 [00:00<00:04, 9.98it/s]
10%|█ | 5/50 [00:00<00:04, 9.95it/s]
12%|█▏ | 6/50 [00:00<00:04, 9.92it/s]
14%|█▍ | 7/50 [00:00<00:04, 9.90it/s]
16%|█▌ | 8/50 [00:00<00:04, 9.88it/s]
18%|█▊ | 9/50 [00:00<00:04, 9.87it/s]
20%|██ | 10/50 [00:01<00:04, 9.87it/s]
22%|██▏ | 11/50 [00:01<00:03, 9.89it/s]
24%|██▍ | 12/50 [00:01<00:03, 9.86it/s]
26%|██▌ | 13/50 [00:01<00:03, 9.86it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.87it/s]
30%|███ | 15/50 [00:01<00:03, 9.86it/s]
32%|███▏ | 16/50 [00:01<00:03, 9.89it/s]
34%|███▍ | 17/50 [00:01<00:03, 9.65it/s]
38%|███▊ | 19/50 [00:01<00:03, 9.89it/s]
42%|████▏ | 21/50 [00:02<00:02, 10.05it/s]
46%|████▌ | 23/50 [00:02<00:02, 10.20it/s]
50%|█████ | 25/50 [00:02<00:02, 10.30it/s]
54%|█████▍ | 27/50 [00:02<00:02, 10.32it/s]
58%|█████▊ | 29/50 [00:02<00:02, 10.36it/s]
62%|██████▏ | 31/50 [00:03<00:01, 10.37it/s]
66%|██████▌ | 33/50 [00:03<00:01, 10.38it/s]
70%|███████ | 35/50 [00:03<00:01, 10.38it/s]
74%|███████▍ | 37/50 [00:03<00:01, 10.37it/s]
78%|███████▊ | 39/50 [00:03<00:01, 10.38it/s]
82%|████████▏ | 41/50 [00:04<00:00, 10.39it/s]
86%|████████▌ | 43/50 [00:04<00:00, 10.37it/s]
90%|█████████ | 45/50 [00:04<00:00, 10.37it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.42it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.35it/s]
100%|██████████| 50/50 [00:04<00:00, 10.19it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.34it/s]
8%|▊ | 4/50 [00:00<00:04, 10.39it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.37it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.42it/s]
20%|██ | 10/50 [00:00<00:03, 10.41it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.42it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.42it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.39it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.38it/s]
40%|████ | 20/50 [00:01<00:02, 10.42it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.43it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.41it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.41it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.44it/s]
60%|██████ | 30/50 [00:02<00:01, 10.47it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.47it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.49it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.46it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.44it/s]
80%|████████ | 40/50 [00:03<00:00, 10.44it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.42it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.41it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.44it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.47it/s]
100%|██████████| 50/50 [00:04<00:00, 10.47it/s]
100%|██████████| 50/50 [00:04<00:00, 10.43it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.17it/s]
8%|▊ | 4/50 [00:00<00:04, 10.26it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.31it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.37it/s]
20%|██ | 10/50 [00:00<00:03, 10.43it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.39it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.38it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.37it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.37it/s]
40%|████ | 20/50 [00:01<00:02, 10.39it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.39it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.35it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.31it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.31it/s]
60%|██████ | 30/50 [00:02<00:01, 10.26it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.26it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.27it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.30it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.32it/s]
80%|████████ | 40/50 [00:03<00:00, 10.33it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.33it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.33it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.35it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.36it/s]
100%|██████████| 50/50 [00:04<00:00, 10.36it/s]
100%|██████████| 50/50 [00:04<00:00, 10.34it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.33it/s]
8%|▊ | 4/50 [00:00<00:04, 10.26it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.32it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.32it/s]
20%|██ | 10/50 [00:00<00:03, 10.33it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.35it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.29it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.28it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.30it/s]
40%|████ | 20/50 [00:01<00:02, 10.32it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.36it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.38it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.39it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.40it/s]
60%|██████ | 30/50 [00:02<00:01, 10.42it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.43it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.44it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.43it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.44it/s]
80%|████████ | 40/50 [00:03<00:00, 10.48it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.47it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.48it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.49it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.41it/s]
100%|██████████| 50/50 [00:04<00:00, 10.40it/s]
100%|██████████| 50/50 [00:04<00:00, 10.39it/s]
0%| | 0/50 [00:00<?, ?it/s]
2%|▏ | 1/50 [00:00<00:04, 9.85it/s]
4%|▍ | 2/50 [00:00<00:04, 9.87it/s]
6%|▌ | 3/50 [00:00<00:04, 9.91it/s]
8%|▊ | 4/50 [00:00<00:04, 9.93it/s]
10%|█ | 5/50 [00:00<00:04, 9.85it/s]
14%|█▍ | 7/50 [00:00<00:04, 9.86it/s]
16%|█▌ | 8/50 [00:00<00:04, 9.88it/s]
18%|█▊ | 9/50 [00:00<00:04, 9.87it/s]
20%|██ | 10/50 [00:01<00:04, 9.89it/s]
22%|██▏ | 11/50 [00:01<00:03, 9.82it/s]
24%|██▍ | 12/50 [00:01<00:03, 9.86it/s]
26%|██▌ | 13/50 [00:01<00:03, 9.81it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.84it/s]
30%|███ | 15/50 [00:01<00:03, 9.88it/s]
32%|███▏ | 16/50 [00:01<00:03, 9.88it/s]
34%|███▍ | 17/50 [00:01<00:03, 9.86it/s]
36%|███▌ | 18/50 [00:01<00:03, 9.83it/s]
38%|███▊ | 19/50 [00:01<00:03, 9.84it/s]
40%|████ | 20/50 [00:02<00:03, 9.88it/s]
42%|████▏ | 21/50 [00:02<00:02, 9.85it/s]
44%|████▍ | 22/50 [00:02<00:02, 9.81it/s]
46%|████▌ | 23/50 [00:02<00:02, 9.83it/s]
48%|████▊ | 24/50 [00:02<00:02, 9.79it/s]
50%|█████ | 25/50 [00:02<00:02, 9.83it/s]
52%|█████▏ | 26/50 [00:02<00:02, 9.86it/s]
56%|█████▌ | 28/50 [00:02<00:02, 9.85it/s]
58%|█████▊ | 29/50 [00:02<00:02, 9.61it/s]
62%|██████▏ | 31/50 [00:03<00:01, 9.89it/s]
66%|██████▌ | 33/50 [00:03<00:01, 10.11it/s]
70%|███████ | 35/50 [00:03<00:01, 10.28it/s]
74%|███████▍ | 37/50 [00:03<00:01, 10.28it/s]
78%|███████▊ | 39/50 [00:03<00:01, 10.36it/s]
82%|████████▏ | 41/50 [00:04<00:00, 10.43it/s]
86%|████████▌ | 43/50 [00:04<00:00, 10.46it/s]
90%|█████████ | 45/50 [00:04<00:00, 10.48it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.47it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.48it/s]
100%|██████████| 50/50 [00:04<00:00, 10.09it/s]
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/synaptic_intelligence.py:65: UserWarning: The Synaptic Intelligence plugin is in an alpha stage and is not perfectly aligned with the paper implementation. Please use at your own risk!
warnings.warn(
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.synaptic_intelligence.SynapticIntelligencePlugin object at 0x7f5367af7b20> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.SynapticIntelligence object at 0x7f5367af7310>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/synaptic_intelligence.py:65: UserWarning: The Synaptic Intelligence plugin is in an alpha stage and is not perfectly aligned with the paper implementation. Please use at your own risk!
warnings.warn(
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.synaptic_intelligence.SynapticIntelligencePlugin object at 0x7f536803c280> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.SynapticIntelligence object at 0x7f536803cbb0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/evaluation.py:85: UserWarning: No benchmark provided to the evaluation plugin. Metrics may be computed on inconsistent portion of streams, use at your own risk.
warnings.warn(
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lr_scheduling.LRSchedulerPlugin object at 0x7f5367b41e50> implements incompatible callbacks for template <avalanche.training.supervised.icarl.ICaRL object at 0x7f5367b41730>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.agem.AGEMPlugin object at 0x7f5367a533a0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.AGEM object at 0x7f5367a53460>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.agem.AGEMPlugin object at 0x7f5367d1c160> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.AGEM object at 0x7f5367d1c670>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.cope.CoPEPlugin object at 0x7f52f80d3850> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.Naive object at 0x7f52f80d3e50>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/evaluation.py:85: UserWarning: No benchmark provided to the evaluation plugin. Metrics may be computed on inconsistent portion of streams, use at your own risk.
warnings.warn(
/home/acossu/reproducible-continual-learning/strategies/dslda/experiment.py:59: UserWarning: The Deep SLDA example is not perfectly aligned with the paper implementation since it does not use a base initialization phase and instead starts streming from pre-trained weights. Performance should still match.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.ewc.EWCPlugin object at 0x7f52f77ff370> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.EWC object at 0x7f52f77ff2e0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gdumb.GDumbPlugin object at 0x7f530411c0a0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.GDumb object at 0x7f530411cc40>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gem.GEMPlugin object at 0x7f531d1e2850> implements incompatible callbacks for template <strategies.gem.experiment.GEM_reduced object at 0x7f531d1e20d0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gem.GEMPlugin object at 0x7f53040f18b0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.GEM object at 0x7f53040f10d0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/evaluation.py:85: UserWarning: No benchmark provided to the evaluation plugin. Metrics may be computed on inconsistent portion of streams, use at your own risk.
warnings.warn(
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.gss_greedy.GSS_greedyPlugin object at 0x7f533a13eb20> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.GSS_greedy object at 0x7f533a13ea90>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/plugins/evaluation.py:85: UserWarning: No benchmark provided to the evaluation plugin. Metrics may be computed on inconsistent portion of streams, use at your own risk.
warnings.warn(
/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lr_scheduling.LRSchedulerPlugin object at 0x7f531d1e2940> implements incompatible callbacks for template <avalanche.training.supervised.icarl.ICaRL object at 0x7f53666dffa0>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lwf.LwFPlugin object at 0x7f53040f18e0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.LwF object at 0x7f53040f1100>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lwf.LwFPlugin object at 0x7f5367b52850> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.LwF object at 0x7f5367b522e0>. This may result in errors.
warnings.warn(
./home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.lwf.LwFPlugin object at 0x7f52f84f6370> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.LwF object at 0x7f52f84f6f70>. This may result in errors.
warnings.warn(
F/home/acossu/miniconda3/envs/repr/lib/python3.8/site-packages/avalanche/training/templates/base.py:205: UserWarning: Plugin <avalanche.training.plugins.mas.MASPlugin object at 0x7f530411c490> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.MAS object at 0x7f530411ceb0>. This may result in errors.
warnings.warn(
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.35it/s]
8%|▊ | 4/50 [00:00<00:04, 10.48it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.56it/s]
16%|█▌ | 8/50 [00:00<00:03, 10.62it/s]
20%|██ | 10/50 [00:00<00:03, 10.58it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.59it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.63it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.65it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.63it/s]
40%|████ | 20/50 [00:01<00:02, 10.63it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.62it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.62it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.61it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.63it/s]
60%|██████ | 30/50 [00:02<00:01, 10.64it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.69it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.69it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.66it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.61it/s]
80%|████████ | 40/50 [00:03<00:00, 10.60it/s]
84%|████████▍ | 42/50 [00:03<00:00, 10.63it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.65it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.63it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.62it/s]
100%|██████████| 50/50 [00:04<00:00, 10.60it/s]
100%|██████████| 50/50 [00:04<00:00, 10.62it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.45it/s]
8%|▊ | 4/50 [00:00<00:04, 10.48it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.47it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.44it/s]
20%|██ | 10/50 [00:00<00:03, 10.42it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.46it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.49it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.50it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.50it/s]
40%|████ | 20/50 [00:01<00:02, 10.51it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.49it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.52it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.53it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.52it/s]
60%|██████ | 30/50 [00:02<00:01, 10.54it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.48it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.45it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.46it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.45it/s]
80%|████████ | 40/50 [00:03<00:00, 10.46it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.51it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.54it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.54it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.52it/s]
100%|██████████| 50/50 [00:04<00:00, 10.49it/s]
100%|██████████| 50/50 [00:04<00:00, 10.49it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.52it/s]
8%|▊ | 4/50 [00:00<00:04, 10.56it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.57it/s]
16%|█▌ | 8/50 [00:00<00:03, 10.58it/s]
20%|██ | 10/50 [00:00<00:03, 10.58it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.60it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.51it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.53it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.55it/s]
40%|████ | 20/50 [00:01<00:02, 10.54it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.53it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.57it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.59it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.54it/s]
60%|██████ | 30/50 [00:02<00:01, 10.58it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.57it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.53it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.50it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.50it/s]
80%|████████ | 40/50 [00:03<00:00, 10.51it/s]
84%|████████▍ | 42/50 [00:03<00:00, 10.52it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.52it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.53it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.52it/s]
100%|██████████| 50/50 [00:04<00:00, 10.50it/s]
100%|██████████| 50/50 [00:04<00:00, 10.54it/s]
0%| | 0/50 [00:00<?, ?it/s]
2%|▏ | 1/50 [00:00<00:05, 9.72it/s]
4%|▍ | 2/50 [00:00<00:04, 9.77it/s]
6%|▌ | 3/50 [00:00<00:04, 9.85it/s]
8%|▊ | 4/50 [00:00<00:04, 9.87it/s]
10%|█ | 5/50 [00:00<00:04, 9.87it/s]
14%|█▍ | 7/50 [00:00<00:04, 9.97it/s]
16%|█▌ | 8/50 [00:00<00:04, 9.90it/s]
18%|█▊ | 9/50 [00:00<00:04, 9.84it/s]
20%|██ | 10/50 [00:01<00:04, 9.83it/s]
24%|██▍ | 12/50 [00:01<00:03, 9.91it/s]
26%|██▌ | 13/50 [00:01<00:03, 9.83it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.84it/s]
30%|███ | 15/50 [00:01<00:03, 9.82it/s]
32%|███▏ | 16/50 [00:01<00:03, 9.83it/s]
34%|███▍ | 17/50 [00:01<00:03, 9.82it/s]
36%|███▌ | 18/50 [00:01<00:03, 9.83it/s]
38%|███▊ | 19/50 [00:01<00:03, 9.80it/s]
40%|████ | 20/50 [00:02<00:03, 9.80it/s]
42%|████▏ | 21/50 [00:02<00:02, 9.81it/s]
44%|████▍ | 22/50 [00:02<00:02, 9.81it/s]
46%|████▌ | 23/50 [00:02<00:02, 9.83it/s]
48%|████▊ | 24/50 [00:02<00:02, 9.81it/s]
50%|█████ | 25/50 [00:02<00:02, 9.81it/s]
52%|█████▏ | 26/50 [00:02<00:02, 9.79it/s]
54%|█████▍ | 27/50 [00:02<00:02, 9.80it/s]
56%|█████▌ | 28/50 [00:02<00:02, 9.81it/s]
58%|█████▊ | 29/50 [00:02<00:02, 9.85it/s]
60%|██████ | 30/50 [00:03<00:02, 9.82it/s]
62%|██████▏ | 31/50 [00:03<00:01, 9.81it/s]
64%|██████▍ | 32/50 [00:03<00:01, 9.81it/s]
66%|██████▌ | 33/50 [00:03<00:01, 9.80it/s]
68%|██████▊ | 34/50 [00:03<00:01, 9.82it/s]
70%|███████ | 35/50 [00:03<00:01, 9.81it/s]
72%|███████▏ | 36/50 [00:03<00:01, 9.83it/s]
74%|███████▍ | 37/50 [00:03<00:01, 9.83it/s]
76%|███████▌ | 38/50 [00:03<00:01, 9.85it/s]
78%|███████▊ | 39/50 [00:03<00:01, 9.81it/s]
80%|████████ | 40/50 [00:04<00:01, 9.83it/s]
82%|████████▏ | 41/50 [00:04<00:00, 9.85it/s]
84%|████████▍ | 42/50 [00:04<00:00, 9.87it/s]
86%|████████▌ | 43/50 [00:04<00:00, 9.80it/s]
88%|████████▊ | 44/50 [00:04<00:00, 9.82it/s]
90%|█████████ | 45/50 [00:04<00:00, 9.84it/s]
92%|█████████▏| 46/50 [00:04<00:00, 9.88it/s]
94%|█████████▍| 47/50 [00:04<00:00, 9.85it/s]
96%|█████████▌| 48/50 [00:04<00:00, 9.85it/s]
98%|█████████▊| 49/50 [00:04<00:00, 9.82it/s]
100%|██████████| 50/50 [00:05<00:00, 9.85it/s]
100%|██████████| 50/50 [00:05<00:00, 9.83it/s]
0%| | 0/50 [00:00<?, ?it/s]
2%|▏ | 1/50 [00:00<00:04, 9.87it/s]
4%|▍ | 2/50 [00:00<00:05, 9.50it/s]
6%|▌ | 3/50 [00:00<00:04, 9.69it/s]
8%|▊ | 4/50 [00:00<00:04, 9.80it/s]
10%|█ | 5/50 [00:00<00:04, 9.84it/s]
12%|█▏ | 6/50 [00:00<00:04, 9.88it/s]
14%|█▍ | 7/50 [00:00<00:04, 9.89it/s]
18%|█▊ | 9/50 [00:00<00:04, 9.93it/s]
20%|██ | 10/50 [00:01<00:04, 9.95it/s]
22%|██▏ | 11/50 [00:01<00:03, 9.96it/s]
26%|██▌ | 13/50 [00:01<00:03, 9.98it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.97it/s]
30%|███ | 15/50 [00:01<00:03, 9.96it/s]
34%|███▍ | 17/50 [00:01<00:03, 9.98it/s]
38%|███▊ | 19/50 [00:01<00:03, 10.00it/s]
42%|████▏ | 21/50 [00:02<00:02, 10.01it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.00it/s]
46%|████▌ | 23/50 [00:02<00:02, 9.97it/s]
48%|████▊ | 24/50 [00:02<00:02, 9.97it/s]
50%|█████ | 25/50 [00:02<00:02, 9.93it/s]
54%|█████▍ | 27/50 [00:02<00:02, 9.97it/s]
58%|█████▊ | 29/50 [00:02<00:02, 10.00it/s]
62%|██████▏ | 31/50 [00:03<00:01, 10.04it/s]
66%|██████▌ | 33/50 [00:03<00:01, 9.98it/s]
70%|███████ | 35/50 [00:03<00:01, 10.02it/s]
74%|███████▍ | 37/50 [00:03<00:01, 9.99it/s]
76%|███████▌ | 38/50 [00:03<00:01, 9.98it/s]
80%|████████ | 40/50 [00:04<00:00, 10.00it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.01it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.00it/s]
90%|█████████ | 45/50 [00:04<00:00, 9.99it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.04it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.05it/s]
100%|██████████| 50/50 [00:05<00:00, 9.98it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.07it/s]
8%|▊ | 4/50 [00:00<00:04, 10.03it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.04it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.06it/s]
20%|██ | 10/50 [00:00<00:03, 10.05it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.07it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.04it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.03it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.06it/s]
40%|████ | 20/50 [00:01<00:03, 9.99it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.04it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.08it/s]
52%|█████▏ | 26/50 [00:02<00:02, 10.03it/s]
56%|█████▌ | 28/50 [00:02<00:02, 10.04it/s]
60%|██████ | 30/50 [00:02<00:01, 10.05it/s]
64%|██████▍ | 32/50 [00:03<00:01, 10.08it/s]
68%|██████▊ | 34/50 [00:03<00:01, 10.07it/s]
72%|███████▏ | 36/50 [00:03<00:01, 10.03it/s]
76%|███████▌ | 38/50 [00:03<00:01, 10.03it/s]
80%|████████ | 40/50 [00:03<00:00, 10.07it/s]
84%|████████▍ | 42/50 [00:04<00:00, 10.04it/s]
88%|████████▊ | 44/50 [00:04<00:00, 10.04it/s]
92%|█████████▏| 46/50 [00:04<00:00, 10.01it/s]
96%|█████████▌| 48/50 [00:04<00:00, 10.03it/s]
100%|██████████| 50/50 [00:04<00:00, 10.02it/s]
100%|██████████| 50/50 [00:04<00:00, 10.04it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 9.94it/s]
8%|▊ | 4/50 [00:00<00:04, 9.99it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.04it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.06it/s]
20%|██ | 10/50 [00:00<00:03, 10.05it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.06it/s]
28%|██▊ | 14/50 [00:01<00:03, 10.07it/s]
32%|███▏ | 16/50 [00:01<00:03, 10.03it/s]
36%|███▌ | 18/50 [00:01<00:03, 10.01it/s]
40%|████ | 20/50 [00:01<00:02, 10.04it/s]
44%|████▍ | 22/50 [00:02<00:02, 10.03it/s]
48%|████▊ | 24/50 [00:02<00:02, 10.00it/s]
50%|█████ | 25/50 [00:02<00:02, 10.00it/s]
54%|█████▍ | 27/50 [00:02<00:02, 10.03it/s]
58%|█████▊ | 29/50 [00:02<00:02, 10.06it/s]
62%|██████▏ | 31/50 [00:03<00:01, 10.05it/s]
66%|██████▌ | 33/50 [00:03<00:01, 10.05it/s]
70%|███████ | 35/50 [00:03<00:01, 10.07it/s]
74%|███████▍ | 37/50 [00:03<00:01, 10.05it/s]
78%|███████▊ | 39/50 [00:03<00:01, 10.05it/s]
82%|████████▏ | 41/50 [00:04<00:00, 10.03it/s]
86%|████████▌ | 43/50 [00:04<00:00, 10.02it/s]
90%|█████████ | 45/50 [00:04<00:00, 9.99it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.06it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.08it/s]
100%|██████████| 50/50 [00:04<00:00, 10.04it/s]
0%| | 0/50 [00:00<?, ?it/s]
4%|▍ | 2/50 [00:00<00:04, 10.01it/s]
8%|▊ | 4/50 [00:00<00:04, 10.02it/s]
12%|█▏ | 6/50 [00:00<00:04, 10.05it/s]
16%|█▌ | 8/50 [00:00<00:04, 10.06it/s]
20%|██ | 10/50 [00:00<00:03, 10.06it/s]
24%|██▍ | 12/50 [00:01<00:03, 10.04it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.94it/s]
30%|███ | 15/50 [00:01<00:03, 9.93it/s]
32%|███▏ | 16/50 [00:01<00:03, 9.93it/s]
36%|███▌ | 18/50 [00:01<00:03, 9.99it/s]
38%|███▊ | 19/50 [00:01<00:03, 9.94it/s]
42%|████▏ | 21/50 [00:02<00:02, 10.00it/s]
46%|████▌ | 23/50 [00:02<00:02, 9.95it/s]
48%|████▊ | 24/50 [00:02<00:02, 9.93it/s]
50%|█████ | 25/50 [00:02<00:02, 9.91it/s]
54%|█████▍ | 27/50 [00:02<00:02, 10.02it/s]
58%|█████▊ | 29/50 [00:02<00:02, 10.00it/s]
62%|██████▏ | 31/50 [00:03<00:01, 9.99it/s]
66%|██████▌ | 33/50 [00:03<00:01, 10.00it/s]
70%|███████ | 35/50 [00:03<00:01, 10.01it/s]
74%|███████▍ | 37/50 [00:03<00:01, 10.03it/s]
78%|███████▊ | 39/50 [00:03<00:01, 10.04it/s]
82%|████████▏ | 41/50 [00:04<00:00, 10.06it/s]
86%|████████▌ | 43/50 [00:04<00:00, 10.05it/s]
90%|█████████ | 45/50 [00:04<00:00, 10.04it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.06it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.06it/s]
100%|██████████| 50/50 [00:04<00:00, 10.01it/s]
0%| | 0/50 [00:00<?, ?it/s]
2%|▏ | 1/50 [00:00<00:04, 9.86it/s]
4%|▍ | 2/50 [00:00<00:04, 9.84it/s]
8%|▊ | 4/50 [00:00<00:04, 9.85it/s]
10%|█ | 5/50 [00:00<00:04, 9.88it/s]
12%|█▏ | 6/50 [00:00<00:04, 9.89it/s]
14%|█▍ | 7/50 [00:00<00:04, 9.91it/s]
16%|█▌ | 8/50 [00:00<00:04, 9.88it/s]
18%|█▊ | 9/50 [00:00<00:04, 9.90it/s]
20%|██ | 10/50 [00:01<00:04, 9.85it/s]
24%|██▍ | 12/50 [00:01<00:03, 9.90it/s]
26%|██▌ | 13/50 [00:01<00:03, 9.86it/s]
28%|██▊ | 14/50 [00:01<00:03, 9.85it/s]
32%|███▏ | 16/50 [00:01<00:03, 9.93it/s]
34%|███▍ | 17/50 [00:01<00:03, 9.88it/s]
36%|███▌ | 18/50 [00:01<00:03, 9.83it/s]
40%|████ | 20/50 [00:02<00:03, 9.91it/s]
42%|████▏ | 21/50 [00:02<00:03, 9.58it/s]
44%|████▍ | 22/50 [00:02<00:02, 9.59it/s]
46%|████▌ | 23/50 [00:02<00:02, 9.65it/s]
48%|████▊ | 24/50 [00:02<00:02, 9.72it/s]
50%|█████ | 25/50 [00:02<00:02, 9.79it/s]
52%|█████▏ | 26/50 [00:02<00:02, 9.83it/s]
54%|█████▍ | 27/50 [00:02<00:02, 9.86it/s]
56%|█████▌ | 28/50 [00:02<00:02, 9.89it/s]
60%|██████ | 30/50 [00:03<00:02, 9.94it/s]
62%|██████▏ | 31/50 [00:03<00:01, 9.95it/s]
64%|██████▍ | 32/50 [00:03<00:01, 9.95it/s]
66%|██████▌ | 33/50 [00:03<00:01, 9.96it/s]
68%|██████▊ | 34/50 [00:03<00:01, 9.95it/s]
70%|███████ | 35/50 [00:03<00:01, 9.96it/s]
72%|███████▏ | 36/50 [00:03<00:01, 9.95it/s]
74%|███████▍ | 37/50 [00:03<00:01, 9.93it/s]
78%|███████▊ | 39/50 [00:03<00:01, 9.99it/s]
82%|████████▏ | 41/50 [00:04<00:00, 10.04it/s]
86%|████████▌ | 43/50 [00:04<00:00, 10.04it/s]
90%|█████████ | 45/50 [00:04<00:00, 10.05it/s]
94%|█████████▍| 47/50 [00:04<00:00, 10.03it/s]
98%|█████████▊| 49/50 [00:04<00:00, 10.00it/s]
100%|██████████| 50/50 [00:05<00:00, 10.00it/s]
100%|██████████| 50/50 [00:05<00:00, 9.90it/s]
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/agem/experiment.py", line 97, in test_scifar100
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.57 != 0.5329411764705884 within 0.03 delta (0.03705882352941159 difference)
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/cope/experiment.py", line 69, in test_smnist
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.93 != 0.2126 within 0.03 delta (0.7174 difference)
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/lwf/experiment.py", line 140, in test_stinyimagenet
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.42 != 0.20620000000000002 within 0.03 delta (0.21379999999999996 difference)
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/agem/experiment.py", line 97, in test_scifar100
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.57 != 0.5329411764705884 within 0.03 delta (0.03705882352941159 difference)
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/cope/experiment.py", line 69, in test_smnist
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.93 != 0.2126 within 0.03 delta (0.7174 difference)
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/iCARL/experiment.py", line 114, in test_scifar100
self.assertAlmostEqual(target_acc, avg_ia, delta=0.03)
AssertionError: 0.62 != 0.4885769444444444 within 0.03 delta (0.1314230555555556 difference)
Traceback (most recent call last):
File "/home/acossu/reproducible-continual-learning/strategies/lwf/experiment.py", line 140, in test_stinyimagenet
self.assertAlmostEqual(target_acc, avg_stream_acc, delta=0.03)
AssertionError: 0.42 != 0.20620000000000002 within 0.03 delta (0.21379999999999996 difference)
Ran 32 tests in 89924.799s
FAILED (failures=7)
Some tests failed. Check server log for details.
I propose to switch the notation. Right now we have:
bug for bugsIMO, this is very confusing at a first glance. If I see a big red cross I immediately think there is a problem with the strategy. In this case, everything is actually correct, we just changed some hyperparameters or tested a new benchmark.
Instead we could have two separate columns:
custom tag if not using any paper.Hello Andrea - I am trying to run the Split_Tiny_ImageNet Experiment but it keep throwing back an error which it seems to be related to the "small_vgg" model.
As Vincenzo noticed last week, we have access to a currently unused VPS. Maybe we should use it to periodically check Avalanche master still reproduces the results?
Some tests failed. Check server log for details.
@AndreaCossu can you update CoPE with the new results? @AlbinSou fixed it in his last PR.
Link to paper: https://arxiv.org/pdf/2112.08654.pdf
I noticed that L2P Strategy is already implemented in avalanche.
Right now we have determinism enabled. This results in slower experiments. I think it would be better to disable it by default and only use it for the unit tests.
Tests ok
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
Running experiments as described in README, an error for missing CLExperience
avalanche-lib Version: 0.4.0a0
When I use:
python experiments/split_mnist/synaptic_intelligence.py
or
python -m unittest tests.SynapticIntelligence.test_smnist
I get something like:
Traceback (most recent call last): File "/home/avabaseline/continual-learning-baselines/experiments/split_mnist/synaptic_intelligence.py", line 7, in <module> from experiments.utils import set_seed, create_default_args File "/home/avabaseline/continual-learning-baselines/experiments/__init__.py", line 1, in <module> from . import split_mnist File "/home/avabaseline/continual-learning-baselines/experiments/split_mnist/__init__.py", line 1, in <module> from .synaptic_intelligence import synaptic_intelligence_smnist File "/home/avabaseline/continual-learning-baselines/experiments/split_mnist/synaptic_intelligence.py", line 7, in <module> from experiments.utils import set_seed, create_default_args File "/home/avabaseline/continual-learning-baselines/experiments/utils.py", line 9, in <module> from avalanche.benchmarks.utils import AvalancheSubset ImportError: cannot import name 'AvalancheSubset' from 'avalanche.benchmarks.utils' (/home/avalanche/avalanche/benchmarks/utils/__init__.py)
avalanche-lib Version: 0.1.0
When I use:
python experiments/split_mnist/synaptic_intelligence.py
or
python -m unittest tests.SynapticIntelligence.test_smnist
I get something like:
Traceback (most recent call last): File "experiments/split_mnist/naive.py", line 1, in <module> import avalanche as avl File "/opt/conda/envs/avabase/lib/python3.7/site-packages/avalanche/__init__.py", line 1, in <module> from avalanche import benchmarks File "/opt/conda/envs/avabase/lib/python3.7/site-packages/avalanche/benchmarks/__init__.py", line 13, in <module> from .classic import * File "/opt/conda/envs/avabase/lib/python3.7/site-packages/avalanche/benchmarks/classic/__init__.py", line 1, in <module> from .ccifar10 import * File "/opt/conda/envs/avabase/lib/python3.7/site-packages/avalanche/benchmarks/classic/ccifar10.py", line 19, in <module> from avalanche.benchmarks.datasets import default_dataset_location File "/opt/conda/envs/avabase/lib/python3.7/site-packages/avalanche/benchmarks/datasets/__init__.py", line 11, in <module> from .torchvision_wrapper import * File "/opt/conda/envs/avabase/lib/python3.7/site-packages/avalanche/benchmarks/datasets/torchvision_wrapper.py", line 40, in <module> from torchvision.datasets import Kinetics400 as torchKinetics400 ImportError: cannot import name 'Kinetics400' from 'torchvision.datasets' (/opt/conda/envs/avabase/lib/python3.7/site-packages/torchvision/datasets/__init__.py)
Okay, something wrong with torch version,
pip install torch==1.9.0 torchvision==0.10.0
Then new problem is :
Traceback (most recent call last): File "/home/avabaseline/continual-learning-baselines/experiments/split_mnist/naive.py", line 7, in <module> from experiments.utils import set_seed, create_default_args File "/home/avabaseline/continual-learning-baselines/experiments/__init__.py", line 1, in <module> from . import split_mnist File "/home/avabaseline/continual-learning-baselines/experiments/split_mnist/__init__.py", line 3, in <module> from .gss import gss_smnist File "/home/avabaseline/continual-learning-baselines/experiments/split_mnist/gss.py", line 3, in <module> from avalanche.benchmarks import CLExperience ImportError: cannot import name 'CLExperience' from 'avalanche.benchmarks' (/opt/conda/envs/ab/lib/python3.9/site-packages/avalanche/benchmarks/__init__.py)
I also try other version of avalanche-lib. And different errors happened, such as: ImportError: cannot import name 'MIRPlugin' from 'avalanche.training.plugins' (/opt/conda/envs/ab/lib/python3.9/site-packages/avalanche/training/plugins/__init__.py)
OMG！
I don't know how to solve it and I'm upset. Maybe I made some silly mistakes. Please forgive me and help me.
Give me some details about correct envinronments plz.
THX!!!
Papers which are used as baseline comparison should be referenced in the code
def synaptic_intelligence_pmnist(override_args=None):
@article{zenke_continual_2017,
title = {Continual {Learning} {Through} {Synaptic} {Intelligence}},
url = {http://arxiv.org/abs/1703.04200},
journal = {arXiv:1703.04200 [cs, q-bio, stat]},
author = {Zenke, Friedemann and Poole, Ben and Ganguli, Surya},
month = jun,
year = {2017},
}
args = create_default_args({'cuda': 0, 'si_lambda': 0.1, 'si_eps': 0.1, 'epochs': 20,
'learning_rate': 0.001, 'train_mb_size': 256, 'seed': 0}, override_args)
Some tests failed. Check server log for details.
Some tests failed. Check server log for details.
First off, I would like to express my gratitude for creating this repository on continual learning baseline and the well put avalanche library. I am currently studying synaptic intelligence and trying to replicate the results, particularly on the SplitMNIST dataset. The page reports 97% accuracy which is described as the average accuracy across all experience after training on the last experience. However, when I run the code, the final accuracy is only 19.27%, which is exactly the same as when I manually evaluate the performance of the trained model against the full MNIST test dataset. Is the 19.27% accuracy correct? Or am I missing something?
All the best,
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.