30B model support about serge HOT 12 CLOSED

Hi guys, I just tested the 30B model, it works fine (with the conversion manually from https://gist.github.com/eiz/828bddec6162a023114ce19146cb2b82) and don't forget the modification in the llama cpp to load a single file for the 30b model.

from serge.

I think you don't need to grab a different tokenizer, I believe they're exactly the same.

You can check it here:
https://huggingface.co/decapoda-research/llama-30b-hf/blob/main/tokenizer.model
https://huggingface.co/decapoda-research/llama-13b-hf/blob/main/tokenizer.model
https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer.model

They all have the same SHA256 hash

from serge.

Yes you right no need, i used same

from serge.

I've made the 30B model available to download but I don't have the hardware to test it. So if someone feels like it, feel free to download it ! Instructions are in the README.

from serge.

I can test it, but it looks like the 7B tokenizer is downloaded. I'm running main at ref 58cf7d0

from serge.

I hacked it locally with this, but it's pretty jank. I think the model should determine the tokenizer.

index 73298b5..d0eafcb 100644
--- a/api/utils/download.py
+++ b/api/utils/download.py
@@ -10,6 +10,7 @@ models_info = {
     "13B": ["Pi3141/alpaca-13B-ggml", "ggml-model-q4_0.bin"],
     "30B": ["Pi3141/alpaca-30B-ggml", "ggml-model-q4_0.bin"],
     "tokenizer": ["decapoda-research/llama-7b-hf", "tokenizer.model"],
+    "30B-tokenizer": ["decapoda-research/llama-30b-hf", "tokenizer.model"],
 }


@@ -21,7 +22,7 @@ def parse_args():
         "model",
         help="Model name",
         nargs="+",
-        choices=["7B", "13B", "30B", "tokenizer"],
+        choices=["7B", "13B", "30B", "tokenizer", "30B-tokenizer"],
     )

     return parser.parse_args()

from serge.

Hi guys, I just tested the 30B model, it works fine (with the conversion manually from https://gist.github.com/eiz/828bddec6162a023114ce19146cb2b82) and don't forget the modification in the llama cpp to load a single file for the 30b model.

Are you sure this is needed ? I was pretty sure the --n_parts argument to llama.cpp could let you handle it without modifying the source.

In serge it's handled there:
https://github.com/nsarrazin/serge/blob/58cf7d0f451035843d8b7e8cac048513fe47e01c/api/utils/generate.py#L47-L48

from serge.

Sorry, you're right, I was on the old method in my head, I had forgotten the features in command line.

from serge.

I hacked it locally with this, but it's pretty jank. I think the model should determine the tokenizer.

index 73298b5..d0eafcb 100644
--- a/api/utils/download.py
+++ b/api/utils/download.py
@@ -10,6 +10,7 @@ models_info = {
     "13B": ["Pi3141/alpaca-13B-ggml", "ggml-model-q4_0.bin"],
     "30B": ["Pi3141/alpaca-30B-ggml", "ggml-model-q4_0.bin"],
     "tokenizer": ["decapoda-research/llama-7b-hf", "tokenizer.model"],
+    "30B-tokenizer": ["decapoda-research/llama-30b-hf", "tokenizer.model"],
 }


@@ -21,7 +22,7 @@ def parse_args():
         "model",
         help="Model name",
         nargs="+",
-        choices=["7B", "13B", "30B", "tokenizer"],
+        choices=["7B", "13B", "30B", "tokenizer", "30B-tokenizer"],
     )

     return parser.parse_args()

Thanks for doing this do you know if it's actually necessary to grab the matching tokenizer.model ?

The instructions here ggerganov/llama.cpp#382 (comment) just mention grabbing a tokenizer, and so I assumed you could use the tokenizer from the 7B repo for all the weights. I'm gonna test for myself it that works still.

And are you able to get any outputs from the 30B model with Serge so far ? @dacamp

from serge.

Thanks @maximeseth

from serge.

Seems like this could be closed then ?

from serge.

I'm closing this, I think it works. If it doesn't I'll reopen it.

from serge.