[BUG]: fine train llama-2-7b-hf prepare data set error , `bos_token` and `eos_token` should be the same with `conversation_template.seps`. about colossalai HOT 2 CLOSED

but meta-Llama-3-8B model correct

root@63733705cb83:/ColossalAI/applications/Colossal-LLaMA# python prepare_sft_dataset.py --data_input_dirs /models/train-data-dir --tokenizer_dir /models/llama-2-13b-hf^Ct_dirs /models/train-data-dir/out --num_spliced_dataset_bins 1
root@63733705cb83:/ColossalAI/applications/Colossal-LLaMA# python prepare_sft_dataset.py --data_input_dirs /models/train-data-dir --tokenizer_dir /models/meta-Llama-3-8B --data_output_dirs /models/train-data-dir/out --num_spliced_dataset_bins 1
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[05/14/24 16:51:56] INFO colossalai - colossalai - INFO: /ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py:101 main
INFO colossalai - colossalai - INFO: Start to process part-0/1 of all original datasets.
Map (num_proc=8): 100%|███████████████████████████████████████████████████████████████████████████████| 8/8 [00:02<00:00, 3.26 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 469.98 examples/s]
[05/14/24 16:51:59] INFO colossalai - colossalai - INFO: /ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py:127 main
INFO colossalai - colossalai - INFO: processing 0 spliced data points for
/models/train-data-dir/out/jsonl/part-00000.jsonl
INFO colossalai - colossalai - INFO: /ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py:133 main
INFO colossalai - colossalai - INFO: Start to save /models/train-data-dir/out/arrow/part-00000
Setting num_proc from 48 back to 1 for the train split to disable multiprocessing as it only contains one shard.
Generating train split: 8 examples [00:00, 967.99 examples/s]
Saving the dataset (8/8 shards): 100%|████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 36.59 examples/s]

from colossalai.

Hi,

As the bos, eos tokens are different for llama2 and llama3, so you need to choose which version of llama you want to process.

We have a parameter to set version of llama here:

The default one is llama3.
Also you will need to change the default conversation template to llama2 at here:

We will fix the issue. Thanks.

from colossalai.