test llama with GLUE about llama HOT 6 CLOSED

The model is just a container of weights, layers, etc. FB has created their own configurations and trained them, that's all. You can use the model, configure it with in your own way (number of heads, layers, embeddings, etc) and then train it. The tricky part here is to train it, because so far they have not provided any code to do that. I don't expect it to be a problem though and I think you could use similar approach as it was used for training other models (like GPT-2) since the architecture is pretty much the same here.

Re the model size - it is hard to answer your questions, because they are too generic. I think they are use cases where 40M is good enough (even much smaller models might be). With a single 8GB GPU, you could train it in a few days I think. But again, it depends on different factors (the training dataset size, parameters, even your implementation). I don't have trained my own LLAMA model at this moment, so nothing to public yet :)

from llama.

You won't be able to train it by yourself. It would take thousands of years on a desktop PC.

There may be a way to train it using a distributed network of computers. For example splitting a batch between a hundred computers and aggregating the results.

Do you own a supercomputer perhaps?

from llama.

@pauldog not necessary. It depends on the model size. If you create a model with ~40M params, you can train and run it locally without any sophisticated setup (6-8GB GPU could be good enough). But you are totally right if we are talking about models x B params.

from llama.

Interesting. I think I must have misunderstood and thought 7B was the minimum you must have.

Will 40M be enough to get something interesting do you think? How long do you think it will take you to train it? Will you release it to the public 😁

Maybe with smaller size you need to pick your training data really well.

from llama.

In case you are still interested in training your own LLaMA-based model, I released my own implementation: https://github.com/chrisociepa/allamo . I tested it on RTX 2060 6GB running training for a model with 90M parameters.

from llama.

For future reference, check the current getting started guides on both llama and llama-recipes repositories

from llama.