Continual pretraining for custom data is not working. Not recognizing TextFiles as a data attribute. about litgpt HOT 3 CLOSED

Hi there,

the only issue I am seeing in your code is that you are missing the --tokenizer_dir. You can set it to

--tokenizer_dir checkpoints/meta-llama/Llama-2-7b-hf

assuming that your checkpoints/meta-llama/Llama-2-7b-hf folder has a tokenizer (it should be downloaded by default).

However, if you are missing the tokenizer_dir there is also a separate error:

Otherwise, the code you have there looks fine. I just tested it and it works without a problem:

My best explanation is that you perhaps have an older version of LitGPT installed that doesn't support TextFiles yet. I recommend installing litgpt from GitHub directly:

pip install -U git+https://github.com/Lightning-AI/litgpt.git

from litgpt.

@rasbt Thanks alot! It's working now. Installing with github directly resolved the issue

from litgpt.

Awesome, that's great to hear!

from litgpt.