Comments (3)
hi, I have modified the model.generate() function for loramoe, and it seems it is working.
You can check the code here : https://github.com/mok0102/LoRAMoE.git -> Try run_test.sh
Still the double check is needed for my modification. Please anyone check my modification and tell me if there is any error.
from loramoe.
@mok0102 @fzp0424 Thank you for your modified code. Because we provide the default value (i.e., None) for the task_type, it can be inferred without modifying. Additionally, we use opencompass to evaluate the fine-tuned models, and we've included profile details in the README.
from loramoe.
我看repo并没有修改generate函数,是在评测时不需要task_type吗?这样会导致报错吧 希望能有eval的代码
os.environ["CUDA_VISIBLE_DEVICES"] = "2" from peft import PeftModel, PeftConfig from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM, LlamaTokenizer output_dir = "./output/0308_debug_format_for_opensource/checkpoint-561" peft_model_path = os.path.join(output_dir, "sft_lora_model") # change checkpoint path peftconfig = PeftConfig.from_pretrained(peft_model_path) model_base = LlamaForCausalLM.from_pretrained(peftconfig.base_model_name_or_path, device_map = "auto", ) tokenizer = LlamaTokenizer.from_pretrained(peftconfig.base_model_name_or_path, add_bos_token = True, add_eos_token = False # always False for inference ) new_model = PeftModel.from_pretrained(model_base, peft_model_path) print("Peft model loaded") import torch def generate_response(prompt, model): encoded_input = tokenizer(prompt, return_tensors="pt", add_special_tokens=True) task_types=0 task_types = torch.tensor(data = task_types) encoded_input["task_types"]=task_types model_inputs = encoded_input.to('cuda') print(model_inputs) generated_ids = model.generate(**model_inputs, max_new_tokens=20, min_new_tokens=1, do_sample=False, pad_token_id=tokenizer.eos_token_id) decoded_output = tokenizer.batch_decode(generated_ids) return decoded_output[0].replace(prompt, "") prompt = """Spanish: Período de validez después de abierto el envase: 10 horas. English:""" generate_response(prompt, new_model)
Error info:
{'input_ids': tensor([[ 1, 10432, 29901, 2431, 29983, 8144, 316, 659, 680, 29920, 11006, 316, 633, 25449, 560, 8829, 559, 29901, 29871, 29896, 29900, 4029, 294, 29889, 13, 24636, 29901]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0'), 'task_types': tensor(0, device='cuda:0')} Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?4edebf64-8b8d-479e-88d4-531631ec5757) --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[4], line 22 17 return decoded_output[0].replace(prompt, "") 19 prompt = """Spanish: Período de validez después de abierto el envase: 10 horas. 20 English:""" ---> 22 generate_response(prompt, new_model) Cell In[4], line 9, in generate_response(prompt, model) 7 model_inputs = encoded_input.to('cuda') 8 print(model_inputs) ----> 9 generated_ids = model.generate(**model_inputs, 10 max_new_tokens=20, 11 min_new_tokens=1, 12 do_sample=False, 13 pad_token_id=tokenizer.eos_token_id) 15 decoded_output = tokenizer.batch_decode(generated_ids) 17 return decoded_output[0].replace(prompt, "") File ~/LoRAMoE/peft/peft_model.py:587, in PeftModelForCausalLM.generate(self, **kwargs) 585 self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation 586 try: --> 587 outputs = self.base_model.generate(**kwargs) 588 except: 589 self.base_model.prepare_inputs_for_generation = self.base_model_prepare_inputs_for_generation ... -> 2349 next_token_logits = outputs.logits[:, -1, :] 2351 # pre-process distribution 2352 next_tokens_scores = logits_processor(input_ids, next_token_logits) AttributeError: 'str' object has no attribute 'logits'
同学,请问你能顺利generate了嘛,想请教一下~谢谢
from loramoe.
Related Issues (15)
- Missing evaluation file HOT 3
- 没有评估流程??? HOT 1
- Potential Bug in Paper or Code HOT 1
- 训练保存下来的模型不是完整的模型,无法使用opencompass评估 HOT 4
- 【bug】代码无法运行 HOT 8
- transformers版本问题 HOT 4
- 训练数据中的"task_type"的作用 HOT 1
- [bug] 代码运行出错!!急!急!非常感谢! HOT 2
- 使用opencompass评估模型出错
- 求指教!!请问论文中的Router模块对应代码的哪部分内容呢 HOT 2
- 请问微调实验需要多少显存
- LoRAMoE和LoRA似乎没有显著差异
- 微调后的模型是否支持使用vLLM进行推理?
- 多模态模型适用吗
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from loramoe.