Question: correct prompts template for llama3-instruct about h2ogpt HOT 9 CLOSED

It works OOTB in newer h2ogpt. Uses chat template with end of turn stopping added.

from h2ogpt.

@pseudotensor Thanks, can you please elaborate more, as I'm not fully understand :) Sorry

from h2ogpt.

e.g. this is sufficient:

python generate.py --base_model=meta-llama/Meta-Llama-3-8B-Instruct

It uses the chat template that is in the HF model repo. This doesn't have a system prompt, so we add that as a pre-conversation.

from h2ogpt.

@pseudotensor Thanks, but if I'm using GGUF version, this not happens.

from h2ogpt.

llama-3 prompting is tricky and so to ensure it is done right, please use latest h2oGPT as of the recent merge and run:

python generate.py --base_model=llama --model_path_llama=https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct.Q5_K_M.gguf?download=true --tokenizer_base_model=meta-llama/Meta-Llama-3-8B-Instruct --max_seq_len=8192

from h2ogpt.

@pseudotensor Thanks, using the latest code and example you provided, it's working fine, can you just provide a bit more details on from where model prompt template is taken in that case and why need to provide --tokenizer_base_model ? Also if using full off-line mode will the --tokenizer_base_model work ?

from h2ogpt.

Yes, once off line then do:

TRANSFORMERS_OFFLINE=1 python generate.py --base_model=llama --model_path_llama=Meta-Llama-3-8B-Instruct.Q5_K_M.gguf --tokenizer_base_model=meta-llama/Meta-Llama-3-8B-Instruct --max_seq_len=8192 --gradio_offline_level=2 --share=False --add_disk_models_to_ui=False

which assumes the model was downloaded to location set by llamacpp_path (default is llamacpp_path folder). This works for offline if previously used the earlier command that got the tokenizer.

from h2ogpt.

The model prompt template is taken from

https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/tokenizer_config.json#L2053

if using that as the tokenizer_base_model

from h2ogpt.

@pseudotensor Thanks a lot !!

from h2ogpt.