Batch inputs get different result to single input for llama model. about transformers HOT 8 CLOSED

According to my debug, the generate mask is not correct. for example, there should be some large negative values in mask, but there is only 0 now.
tensor([[[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]],

    [[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]])

from transformers.

@gante @jianan-gu

from transformers.

Hey @liangan1 !

There are a few things that have to be changed for generation to work properly in batched form. Firstly, tt is recommended to use the left padding side in generation if you are using a decoder-only models. Also, attention mask needs to be passed into the generate if input is batched. Please try the code below to verify that it works :)

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf", padding_side='left')  # left padding for generation
model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
inputs = tokenizer(["how are you?", "what is the best the AI algorithm?"], return_tensors="pt",padding=True)
output = model.generate(**inputs, max_new_tokens=10)  # pass in not only input ids, but also attention mask
out_text = tokenizer.batch_decode(output, skip_special_tokens=True)
print(out_text)

input_ids = tokenizer(["how are you?"], return_tensors="pt",padding=True).input_ids
output = model.generate(input_ids, max_new_tokens=10)
out_text = tokenizer.batch_decode(output, skip_special_tokens=True)
print(out_text)

from transformers.

@zucchini-nlp thanks. why user need to create mask by themself?

from transformers.

@liangan1 you don't have to create it manually. The tokenizer returns attention mask, which should be passed into generate.

inputs = tokenizer(["how are you?", "what is the best the AI algorithm?"], return_tensors="pt",padding=True)
print(inputs.attention_mask)

I will close the issue as resolved. For any further questions it is recommended to ask in the forum 🤗

from transformers.

Thanks for your help.

from transformers.

@liangan1 to complement the answer above: there are a few seemingly innocuous differences that may result in slightly different LLM outputs, such as batching. To understand why it happens (and why it is unavoidable), have a look at this comment :)

from transformers.

@liangan1 to complement the answer above: there are a few seemingly innocuous differences that may result in slightly different LLM outputs, such as batching. To understand why it happens (and why it is unavoidable), have a look at this comment :)

Thanks for your info.

from transformers.