[Bug]: Model "talking to itself" and ignoring `<|im_end|>` about vllm HOT 13 CLOSED

Specifically, you seem to be using the meta-llama/Meta-Llama-3-8B model in a chat context, so I suspect you are querying the /chat/completions API

This model does not have a chat template, so the openai server is going to use a default template which will not work well

Please try with meta-llama/Meta-Llama-3-8B-Instruct

from vllm.

@robertgshaw2-neuralmagic does this ring a bell? any clue?

from vllm.

@eamonnlambda - can you share some sample requests you are sending so I can debug?

from vllm.

@eamonnlambda what's is the eos_token_id in the model's config.json, and in generation_config.json if it exists? The model will only stop if the <|im_end|> token is configured as that.

It looks like you're trying to do chat stuff with a non- chat-tuned model?

from vllm.

• meta-llama/Meta-Llama-3-8B's eos_token is <|end-of-text|>
• https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/tokenizer_config.json#L2054

So I think what is going is that you are sending chat requests, we have the default chat-template (which I think use im_end and then the model is responding as if it were a few shot scenario)

from vllm.

Hi!

I appreciate all the responses here. I just tried the -Instruct model, and it appears to be working better. However, I will note, I originally saw this issue using the exact examples provided in the documentation.

Using facebook/opt-125m, I receive the output:

It's not a problem, people will just be logging out of the lobby on their phone so you can still use it.\n> You'll still be able to access the lobby when you're in the lobby, so you can still use it  ...  ...  ...  ......  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...  ...  ...  ...  ...  ...  ...  ...  ...  ...   ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n

Now, as much as I like games, I would expect a result slightly different from the above for the question being asked. Would it then make sense to modify the documentation to use a model that does not produce spurious output?

Thanks again for everyone dropping in!

from vllm.

Can you post the command you are running so that I can re-create?

from vllm.

Hi!

I appreciate all the responses here. I just tried the -Instruct model, and it appears to be working better. However, I will note, I originally saw this issue using the exact examples provided in the documentation.

Using facebook/opt-125m, I receive the output:

It's not a problem, people will just be logging out of the lobby on their phone so you can still use it.\n> You'll still be able to access the lobby when you're in the lobby, so you can still use it  ...  ...  ...  ......  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...  ...  ...  ...  ...  ...  ...  ...  ...  ...   ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n

Now, as much as I like games, I would expect a result slightly different from the above for the question being asked. Would it then make sense to modify the documentation to use a model that does not produce spurious output?

Thanks again for everyone dropping in!

We can only do as good as the model! opt-125m is not a good model!

from vllm.

Hi!

I appreciate all the responses here. I just tried the -Instruct model, and it appears to be working better. However, I will note, I originally saw this issue using the exact examples provided in the documentation.

Using facebook/opt-125m, I receive the output:

It's not a problem, people will just be logging out of the lobby on their phone so you can still use it.\n> You'll still be able to access the lobby when you're in the lobby, so you can still use it  ...  ...  ...  ......  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...  ...  ...  ...  ...  ...  ...  ...  ...  ...   ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n

Now, as much as I like games, I would expect a result slightly different from the above for the question being asked. Would it then make sense to modify the documentation to use a model that does not produce spurious output?

Thanks again for everyone dropping in!

Which documentation are you referring to?

from vllm.

Certainly!

python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m

curl --location --request POST 'localhost:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "model": "facebook/opt-125m",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Who won the world series in 2020?"
        }
    ]
}'

It should be the same as in the Quickstart page: https://docs.vllm.ai/en/stable/getting_started/quickstart.html#using-openai-chat-api-with-vllm

from vllm.

Certainly!

python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m

curl --location --request POST 'localhost:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "model": "facebook/opt-125m",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Who won the world series in 2020?"
        }
    ]
}'

It should be the same as in the Quickstart page: https://docs.vllm.ai/en/stable/getting_started/quickstart.html#using-openai-chat-api-with-vllm

Thanks - Im going to swap out the model here. Best to use llama-3-instruct for the samples as opt gives garbage in the chat setup

from vllm.

Perfect. This can be safely closed, then. I really appreciate all the help!

In terms of opt-125m not being a good model... I'm not going to disagree with you there!

from vllm.

Sounds good - let me know if you need any other help

from vllm.