zju-vipa / cmi Goto Github PK

View Code? Open in Web Editor NEW

65.0 6.0 16.0 2.63 MB

[IJCAI-2021] Contrastive Model Inversion for Data-Free Knowledge Distillation

Python 99.10% Shell 0.90%

data-free knowledge-distillation model-compression benchmark

cmi's People

Contributors

Stargazers

Watchers

Forkers

zhenyuanlin mornydew vishal-s-p trendingtechnology sharpiless joanyu suqinghang qigongsun kai-wen-yang ljrprocc mldl emmetwilliam esoufleri lhh20000923 dl-kd

cmi's Issues

About running environment

Hi there~

Thank you for the fine work.
Can I know the exact torch version, please? I saw the "requirements.txt" file, but versions weren't specified.
I'm having some issues regarding it.

Thank you

The accuracy of code reproduction

I try to repruduce your impressive work. I just simply run 'bash scripts/cmi/adv_cmi_cifar10_wrn402_wrn161.sh' but get lower accuracy than paper:

Use GPU: 0 for training
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: method: cmi
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: adv: 0.5
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: bn: 1.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: oh: 0.5
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: act: 0.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: balance: 0.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: save_dir: run/adv_cmi_cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cr: 0.8
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cr_T: 0.1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cmi_init: run/cmi-preinverted-wrn402
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: data_root: data
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: teacher: wrn40_2
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: student: wrn16_1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dataset: cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr: 0.1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr_decay_milestones: 25,30,35
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr_g: 0.001
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: T: 20.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: epochs: 40
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: g_steps: 200
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: kd_steps: 2000
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: ep_steps: 2000
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: resume: 
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: evaluate_only: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: batch_size: 128
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: synthesis_batch_size: 256
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: gpu: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: world_size: -1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: rank: -1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dist_url: tcp://224.66.41.62:23456
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dist_backend: nccl
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: multiprocessing_distributed: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: fp16: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: seed: None
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: log_tag: -adv_cmi_cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: workers: 4
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: start_epoch: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: momentum: 0.9
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: weight_decay: 0.0001
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: print_freq: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: pretrained: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: distributed: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: ngpus_per_node: 1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: autocast: <function dummy_ctx at 0x7f61b2802560>
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: logger: <Logger cifar10-wrn40_2-wrn16_1 (DEBUG)>
Files already downloaded and verified
Files already downloaded and verified
CMI dims: 2704
[08/27 20:20:43 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=0 Acc@1=23.6500 Acc@5=76.6300 Loss=4.2118 Lr=0.1000
[08/27 20:24:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=1 Acc@1=31.9100 Acc@5=81.9000 Loss=3.3887 Lr=0.0998
[08/27 20:27:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=2 Acc@1=36.4000 Acc@5=85.6800 Loss=3.0814 Lr=0.0994
[08/27 20:30:02 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=3 Acc@1=42.9800 Acc@5=88.9000 Loss=2.7402 Lr=0.0986
[08/27 20:32:29 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=4 Acc@1=45.4600 Acc@5=91.3300 Loss=2.6151 Lr=0.0976
[08/27 20:34:49 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=5 Acc@1=51.9300 Acc@5=93.0500 Loss=2.3906 Lr=0.0962
[08/27 20:37:04 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=6 Acc@1=52.5100 Acc@5=91.0300 Loss=2.4513 Lr=0.0946
[08/27 20:39:13 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=7 Acc@1=56.1100 Acc@5=94.8800 Loss=2.0773 Lr=0.0926
[08/27 20:41:20 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=8 Acc@1=58.4500 Acc@5=95.4500 Loss=2.1580 Lr=0.0905
[08/27 20:43:24 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=9 Acc@1=64.0700 Acc@5=96.5000 Loss=1.6752 Lr=0.0880
[08/27 20:45:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=10 Acc@1=61.5900 Acc@5=96.7800 Loss=1.9675 Lr=0.0854
[08/27 20:47:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=11 Acc@1=65.8900 Acc@5=96.1100 Loss=1.6139 Lr=0.0825
[08/27 20:49:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=12 Acc@1=66.5500 Acc@5=96.0800 Loss=1.6001 Lr=0.0794
[08/27 20:51:26 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=13 Acc@1=69.8900 Acc@5=97.6900 Loss=1.3662 Lr=0.0761
[08/27 20:53:24 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=14 Acc@1=66.6300 Acc@5=97.1500 Loss=1.5617 Lr=0.0727
[08/27 20:55:21 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=15 Acc@1=72.4200 Acc@5=98.0800 Loss=1.1806 Lr=0.0691
[08/27 20:57:19 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=16 Acc@1=72.5100 Acc@5=98.1300 Loss=1.2018 Lr=0.0655
[08/27 20:59:15 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=17 Acc@1=68.6100 Acc@5=98.2000 Loss=1.5399 Lr=0.0617
[08/27 21:01:11 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=18 Acc@1=74.9700 Acc@5=98.4700 Loss=1.0825 Lr=0.0578
[08/27 21:03:06 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=19 Acc@1=70.5000 Acc@5=97.9700 Loss=1.4037 Lr=0.0539
[08/27 21:05:01 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=20 Acc@1=74.3100 Acc@5=98.4500 Loss=1.1729 Lr=0.0500
[08/27 21:06:55 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=21 Acc@1=76.1000 Acc@5=98.3600 Loss=1.0403 Lr=0.0461
[08/27 21:08:49 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=22 Acc@1=76.4900 Acc@5=98.1800 Loss=1.0029 Lr=0.0422
[08/27 21:10:42 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=23 Acc@1=77.2500 Acc@5=98.5500 Loss=1.0258 Lr=0.0383
[08/27 21:12:34 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=24 Acc@1=76.9900 Acc@5=98.7200 Loss=0.9811 Lr=0.0345
[08/27 21:14:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=25 Acc@1=77.7800 Acc@5=98.5100 Loss=0.9825 Lr=0.0309
[08/27 21:16:19 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=26 Acc@1=77.1600 Acc@5=98.5800 Loss=1.0215 Lr=0.0273
[08/27 21:18:09 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=27 Acc@1=78.0100 Acc@5=98.4900 Loss=0.9878 Lr=0.0239
[08/27 21:20:00 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=28 Acc@1=77.0600 Acc@5=98.7700 Loss=1.0611 Lr=0.0206
[08/27 21:21:52 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=29 Acc@1=78.3700 Acc@5=98.7500 Loss=1.0098 Lr=0.0175
[08/27 21:23:42 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=30 Acc@1=78.0200 Acc@5=98.8700 Loss=1.0001 Lr=0.0146
[08/27 21:25:32 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=31 Acc@1=79.6000 Acc@5=99.0600 Loss=0.8993 Lr=0.0120
[08/27 21:27:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=32 Acc@1=80.0800 Acc@5=99.1700 Loss=0.8516 Lr=0.0095
[08/27 21:29:13 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=33 Acc@1=80.1500 Acc@5=99.1100 Loss=0.8723 Lr=0.0074
[08/27 21:31:03 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=34 Acc@1=81.4100 Acc@5=99.0800 Loss=0.8326 Lr=0.0054
[08/27 21:32:53 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=35 Acc@1=81.4900 Acc@5=99.1500 Loss=0.8322 Lr=0.0038
[08/27 21:34:43 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=36 Acc@1=81.2300 Acc@5=99.1200 Loss=0.8257 Lr=0.0024
[08/27 21:36:32 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=37 Acc@1=81.7300 Acc@5=99.2600 Loss=0.8082 Lr=0.0014
[08/27 21:38:21 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=38 Acc@1=81.6900 Acc@5=99.2500 Loss=0.7939 Lr=0.0006
[08/27 21:40:10 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=39 Acc@1=81.8800 Acc@5=99.2300 Loss=0.7991 Lr=0.0002
[08/27 21:40:10 cifar10-wrn40_2-wrn16_1]: Best: 81.8800

Detail of training teacher model

Hi, thank you for this great job

Can you publish the training details about the teacher model? For example, ResNet34. I can't get a very clear inversion picture using my own trained ResNet18 and 34. Is it because you used special training techniques when training the pre-training model or the hyperparameters of CMI need to be adjusted accordingly. If the latter, can you provide some guidance on how to tune hyperparameters?

Regarding applying BDD100K dataset for teacher as Resnet-50 and my own architecture for student

Greetings,

Is it possible to use your codebase on BDD100K dataset ? I want to use teacher as Resnet-50 and my own architecture for Student.

Thanks

Some questions about the result of DFQ

I ran the code according to the "Scripts" of DFQ.
Teacher - wrn_40_2, Student - wrn_16_1, dataset - CIFAR10
But the result is too much outstanding. The top1 accuracy in testset is more than 92%, which is even better than the result of cmi in the paper. I'm wondering if this is a coincidence or there are something wrong in the code?

Thx!

About TV loss

Hi~
Thank you for this great work.

My question is about the TV loss. Could you give the reason why you took mean while calculating the TV loss?
The paper about that work did not mention the 'mean'. Thank you

CMI/datafree/criterions.py

Line 48 in 9e79fa9

diff3.abs() / 255.0).mean() + (diff4.abs() / 255.0).mean()

Pretrained model about cifar100

Thanks for your impressive work and code. Could you provide pretrained model about cifar100?

Failed to reproduce.

I replayed the code from pre-training and got the following results:

Model	Method	Dataset	top1-accuracy
ResNet34*	Teacher	Cifar-10	93.94（95.70）
wrn_40_2	Teacher	Cifar-10	92.01（94.87）

For DFQ algorithm, the results are as follows:

Model	Data-Free Method	Student-Loss	Generative-Loss	Dataset	top1-accuracy
ResNet34-ResNet18	DFQ（Baseline）	KL	adv+bn+oh	Cifar-10	88.89（94.61）
ResNet34-ResNet18	DFQ（Baseline）	KL	adv+bn+oh	Cifar-100	1.89（77.01）

For the above results, I reproduce DFQ based on your code for some reason. But the results are poor. I would be grateful if you could kindly give me your advice.

super parameters of resnet34 and resnet18 on cifar-10

Can you provide the training super parameters of resnet34 and resnet18 on cifar10/cifar100?

question about pre-inverted data set

This part is not mentioned in the paper. Again, there is no code in the repo to generate this data. Did I miss something?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.