adamdad / knowledgefactor Goto Github PK
View Code? Open in Web Editor NEW[ECCV2022] Factorizing Knowledge in Neural Networks
License: Apache License 2.0
[ECCV2022] Factorizing Knowledge in Neural Networks
License: Apache License 2.0
What does it mean?
What's the formula for it?
Excuse, a question about the derivations of the Variational Bound for Mutual Information is not in the supplementary materials .
Excuse me, are these ckpts pretrained by ImageNet? I trained the wideresnet's teacher model with a pretrained ckpt solely on cifar10 initially and found it started at a accuracy of 50%, so I guess the ckpts aren't pretrained on the datasets of cifar10/100. Is that right?
Nice work and the code. I find some equation errors in the main text.
Excuse me, I wanted to follow this wonderful work. I have successfully conducted KF experiments on cifar10 and
imagenet; however, a problem occurs when running KD experiments.
on cifar10, I run with: python tools/dist_train.py configs/cifar10-kd/wideresnet28-10_resnet18_b128x1
_cifar10.py 3
I get an error:
so I changed line 159 of the /KnowledgeFactor-main/cls/mmcls/models/classifiers/kd.py with:
loss_cls = self.criterionCls(student_logit, gt_label.squeeze(dim=1))
However, training with the modified code achieved more than 96% acc,which is higher than the acc reported in the paper
So I think my modification is wrong, would you please tell me the right way? thank you very much!
Hi,
Great work! When I run the tools/train.py file with the config file I get this following error.
Also, where should I put the checkpoints after downloading them?
Thanks
Traceback (most recent call last):
File "tools/train.py", line 15, in
from mmcls.apis import set_random_seed, train_model
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/apis/init.py", line 2, in
from .inference import inference_model, init_model, show_result_pyplot
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/apis/inference.py", line 10, in
from mmcls.datasets.pipelines import Compose
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/datasets/init.py", line 2, in
from .base_dataset import BaseDataset
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/datasets/base_dataset.py", line 13, in
from mmcls.models.losses import accuracy
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/models/init.py", line 2, in
from .backbones import * # noqa: F401,F403
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/models/backbones/init.py", line 3, in
from .conformer import Conformer
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcls/models/backbones/conformer.py", line 9, in
from mmcv.cnn.bricks.transformer import AdaptivePadding
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcv/cnn/bricks/transformer.py", line 22, in
from mmcv.ops.multi_scale_deform_attn import MultiScaleDeformableAttention # noqa F401
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcv/ops/init.py", line 10, in
from .corner_pool import CornerPool
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcv/ops/corner_pool.py", line 8, in
ext_module = ext_loader.load_ext('_ext', [
File "/home/amir-e/.conda/envs/kf/lib/python3.8/site-packages/mmcv/utils/ext_loader.py", line 15, in load_ext
assert hasattr(ext, fun), f'{fun} miss in module {name}'
AssertionError: top_pool_forward miss in module _ext
Nice work! But a problem occur when i test running :
Pretrained ResNet18 load successfully in the fold configs/Imagenet-kf ,while it shows "Teacher model not loaded โ when I test running with configs/imagenet-kf/wideresnet28-2*.py. Then I catch the exception , following is some of the error.
RuntimeError ('Error(s) in loading state_dict for ModuleDict:\n\tsize mismatch for backbone.layer1.0.conv1.weight: copying a param with shape torch.Size([160, 16, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 16, 3, 3]).\n\tsize mismatch for backbone.layer1.0.conv2.weight: copying a param with shape torch.Size([160, 160, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 32, 3, 3]).\n\tsize mismatch for backbone.layer1.0.bn2.weight: copying a param with shape torch.Size([160]) from checkpoint, the shape in current model is torch.Size([32]).\n\tsize mismatch for backbone.layer1.0.bn2.bias: copying a param with shape torch.Size([160]) from checkpoint, the shape in current model is torch.Size([32]).\n\tsize mismatch for backbone.layer1.0.bn2.running_mean: copying a param with shape torch.Size([160]) from checkpoint, the shape in current model is torch.Size([32]).\n\tsize mismatch for backbone.layer1.0.bn2.running_var: copying a param with shape torch.Size([160]) from checkpoint, the shape in current model is torch.Size([32]).\n\tsize mismatch for backbone.layer1.0.downsample.0.weight: copying a param with shape torch.Size([160, 16, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 16, 1, 1]).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.