Comments (5)
Hi, thanks for having a look at the code. I did not test dual-gpu training, and RN101 indeed takes quite some time on single GPU (~2 weeks). I did not do the effort of implementing multi-gpu support, since I had to use the other available GPUs in our lab for other runs/experiments.
I suspect some changes are needed in the loss. I was planning to look at it anyway in the coming weeks, I'll let you know!
I also plan to release a trained mobilenetv2 with the optimized CUDA code integrated.
from dynconv.
Hi, @thomasverelst
Thanks for your prompt reply and sharing! I have realized your concern about the computational resource, but two weeks is still a fairly long experimental period :).
Furthermore, I have made attempts towards multi-gpu training by simply wrapping the model with torch.nn.DataParallel
, but was stucked in some issues:
- gather the output dict
meta
across GPUs (possibly I have solved this) - the weights of self-constructed tensors here probably cannot be replicated to other GPUs from GPU 0
Looking forward to your good news! Also congratulations on the upcoming MobileNetV2 CUDA code!
from dynconv.
I've pushed a new branch multigpu
. I didn't test training accuracy yet, but it runs. I only had problems with gathering the output dict meta
. I considered subclassing DataParallel
to support meta
but decided to just change the internal working so PyTorch wouldn't complain.
Note that the pretrained checkpoints are different from the master branch (url in README).
from dynconv.
Yeah, it seems to work now. I have successfully run this branch with ResNet-32 on CIFAR for fast prototyping (with matched accuracy and reduced FLOPs). As an additional note, the "FLOPs counting to zero" problem can be solved by modifying the following line
https://github.com/thomasverelst/dynconv/blob/multigpu/classification/main_cifar.py#L204
model = flopscounter.add_flops_counting_methods(model)
to
model = flopscounter.add_flops_counting_methods(model.module)
, due to the DataParallel
wrapping.
from dynconv.
Thanks a lot, that fixed it.
from dynconv.
Related Issues (14)
- A question about soft-mask calculation HOT 2
- question about the sparsity_target HOT 2
- About pose environment HOT 1
- Questions about mask generation HOT 2
- Questions about mask usage in convolution
- 发生异常: RuntimeError CUDA error: the launch timed out and was terminated File "/home/lym/Compare experiment new/classification/main_cifar.py", line 72, in main model = net_module(sparse=args.budget >= 0, pretrained=args.pretrained).to(device=device) File "/home/lym/Compare experiment new/classification/main_cifar.py", line 232, in <module> main()
- Mask calculation HOT 6
- Does the cuda version support standard convolutions? HOT 2
- About the "Classification with efficient sparse MobileNetV2" HOT 2
- Ponder_Cost_Plotting
- Training on Google Colab HOT 1
- /annot/valid.json is missing HOT 3
- license HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dynconv.