tokenizer.py about litgpt HOT 2 CLOSED

Hi there,
thanks for reporting this. If the Hub does not provide a tokenizer.model file, it should use the tokenizers library instead:

In the case of Llama 3, only the tokenizer.json is provided, so it should not use sentencepiece:

I was trying this in a fresh environment

git clone https://github.com/Lightning-AI/litgpt.git   
cd litgpt

pip install -e ".[all]"
litgpt download --repo_id meta-llama/Meta-Llama-3-8B-Instruct --access_token ...

pip uninstall sentencepiece 
litgpt chat --checkpoint_dir checkpoints/meta-llama/Meta-Llama-3-8B-Instruct 

and it worked without problem.

I was using tokenizers version '0.19.1', perhaps you have an older version installed? Could you try to upgrade it via
pip install tokenizers -U and give it another try?

from litgpt.

Let us know if you still have problems here, happy to reopen and revisit!

from litgpt.