微调ChatGLM
基于清华的 ChatGLM-6B + Alpaca 方式进行finetune.
数据集: 中文alpaca
pip install -r requirements.txt
cd data
git clone https://github.com/carbonz0/alpaca-chinese-dataset
转化alpaca数据集为按行存储的Intruct格式数据
python cover_alpaca2jsonl.py然后把数据划分为train.txt和valid.txt，保存在 ./data/example/路径下
bash scripts/finetune.sh# 要先把文件中的 "output/your_model_dir"替换为实际模型路径
python infer.py
如果有疑问和建议，欢迎加入我们的大模型交流群
转换出来的JSON文件格式有问题，请问是怎么解决的呢？
大佬，请问有做过微调效果的验证吗？
Traceback (most recent call last):
File "run_clm.py", line 564, in
main()
File "run_clm.py", line 512, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/opt/conda/envs/GLM_instruct/lib/python3.8/site-packages/transformers/trainer.py", line 1633, in train
return inner_training_loop(
File "/opt/conda/envs/GLM_instruct/lib/python3.8/site-packages/transformers/trainer.py", line 1902, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/opt/conda/envs/GLM_instruct/lib/python3.8/site-packages/transformers/trainer.py", line 2645, in training_step
loss = self.compute_loss(model, inputs)
File "/opt/conda/envs/GLM_instruct/lib/python3.8/site-packages/transformers/trainer.py", line 2677, in compute_loss
outputs = model(**inputs)
File "/opt/conda/envs/GLM_instruct/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/[email protected]/lck/ChatGLM-Instruct-Tuning/modeling_chatglm.py", line 1033, in forward
transformer_outputs = self.transformer(
File "/opt/conda/envs/GLM_instruct/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/[email protected]/lck/ChatGLM-Instruct-Tuning/modeling_chatglm.py", line 836, in forward
mask_position = seq.index(mask_token)
ValueError: 150001 is not in list
wandb: Waiting for W&B process to finish... (failed 1).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /workspace/[email protected]/lck/ChatGLM-Instruct-Tuning/wandb/offline-run-20230408_040222-i5caj5n7
wandb: Find logs at: ./wandb/offline-run-20230408_040222-i5caj5n7/logs
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--train_file ./data/train.txt
--validation_file ./data/valid.txt
--max_seq_length 256 \
减小了batch_size 还是不行QAQ
您好，基于您的代码，我在本地数据上进行了微调，但调用本地微调之后的模型后 chat 输出的答复都是空字符串，想请问一下是有什么特殊的调用方法吗？
请问你们在训练时用什么显卡训练的，我用A100 40G会爆显存
模型finetuning的时候出现了精度报错，这个需要怎么修改，--torch_dtype 设置为auto
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.