Comments (6)
Hey,
Thanks!
You listed an AMI that includes the libraries, but seems it can be run on EC2 instances that don't have a GPU.
Just to confirm, could you say what EC2 instance you have to replicate this issue?
And try running nvidia-smi
to be sure you have a GPU and it's working correctly.
from backprop.
Hey Kristo,
The details of the EC2 Instance are as follows:
AMI Name: Deep Learning Base AMI (Ubuntu 18.04) Version 37.0
Instance Type: t2.medium
I tried the nvidia-smi command and got the following response:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Thanks.
from backprop.
Seems like this is the problem. The t2.medium
instance doesn't have a GPU.
You need to go for p3
, g3
, g4
, or p4
instance types.
https://docs.aws.amazon.com/dlami/latest/devguide/gpu.html
Cheapest one I found, as an example, is g4dn.xlarge
.
from backprop.
Hi Kristo,
Thank you for sharing this. I was able to run the fine-tune example of g4dn.xlarge, and could successfully train and save the model.
Now, the question is, how can I use the trained model on a non-cuda device?
I've copied the model folder to my local development machine (MacBook Air - Non Cuda), and I get the following error when I load the fine-tuned model.
model = backprop.TextGeneration("genQA")
Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
I've tried the following as well:
model = backprop.TextGeneration("genQA", device="cpu")
but the error persists.
Note: I was able to load and use the model on the CUDA machine.
Thanks for being patient with me.
from backprop.
Glad you could get finetuning to work!
Did you use backprop.save
or model.save
to save the model? If so, there's an oversight on our part where it does not put the model on CPU before serializing it.
As to fixing this, doing the suggested torch.load
with map_location
should work in theory, but it didn't for me (relevant issue on torch).
So we can't include the loading as a fix in the library.
Currently you could do this:
- Load the model on a CUDA machine with
model = backprop.TextGeneration("genQA", device="cpu")
- this loads the model on the CUDA device and puts it on CPU. - Save the model again with
model.save("genQA")
- Now you should be able to load the model on a CPU only machine
We'll include the fix of always saving on CPU for better protability in a future release.
Thanks!
from backprop.
Worked like a charm.
Thank you so much!
from backprop.
Related Issues (9)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from backprop.