Comments (6)
The model is just a container of weights, layers, etc. FB has created their own configurations and trained them, that's all. You can use the model, configure it with in your own way (number of heads, layers, embeddings, etc) and then train it. The tricky part here is to train it, because so far they have not provided any code to do that. I don't expect it to be a problem though and I think you could use similar approach as it was used for training other models (like GPT-2) since the architecture is pretty much the same here.
Re the model size - it is hard to answer your questions, because they are too generic. I think they are use cases where 40M is good enough (even much smaller models might be). With a single 8GB GPU, you could train it in a few days I think. But again, it depends on different factors (the training dataset size, parameters, even your implementation). I don't have trained my own LLAMA model at this moment, so nothing to public yet :)
from llama.
You won't be able to train it by yourself. It would take thousands of years on a desktop PC.
There may be a way to train it using a distributed network of computers. For example splitting a batch between a hundred computers and aggregating the results.
Do you own a supercomputer perhaps?
from llama.
@pauldog not necessary. It depends on the model size. If you create a model with ~40M params, you can train and run it locally without any sophisticated setup (6-8GB GPU could be good enough). But you are totally right if we are talking about models x B params.
from llama.
Interesting. I think I must have misunderstood and thought 7B was the minimum you must have.
Will 40M be enough to get something interesting do you think? How long do you think it will take you to train it? Will you release it to the public 😁
Maybe with smaller size you need to pick your training data really well.
from llama.
In case you are still interested in training your own LLaMA-based model, I released my own implementation: https://github.com/chrisociepa/allamo . I tested it on RTX 2060 6GB running training for a model with 90M parameters.
from llama.
For future reference, check the current getting started guides on both llama and llama-recipes repositories
from llama.
Related Issues (20)
- "Link unavailable" in Meta AI response for asking Meta Website HOT 6
- Your request to access this repo has been rejected by the repo's authors.
- Research dedicated license? HOT 2
- Bug HOT 1
- "$CPU_ARCH" not found HOT 2
- Unable to download LLAMA models from https://llama.meta.com/llama-downloads HOT 1
- Download.sh does nothing HOT 2
- No special tokens added in tokenizer
- Downlaod.sh is throwing 403 Foribdeen error, when using a freshly generated URL/token HOT 2
- Getting 400 error on https://llama3-1.llamameta.net/Meta-Llama-3.1-405B-MP8/consolidated.00.pth HOT 1
- How to to run Meta-Llama-3.1-70B-Instruct on the MATH TEST HOT 1
- AnT1nG-Meta-llana
- Llama
- A problem with tokenizer.model from HuggingFace HOT 2
- LLaMA 3.1 Model with Image Input is Faster than Text-only Input
- Llama 3.1: The output text is truncated HOT 2
- Access to checkpoints
- Do you open source of your internal evaluations library ?
- Meta AI issue to append duplicated messages after the answer when questions have context
- Meta-Llama-3.1-70B-Instruct does not appear to have a file named config.json HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llama.