Comments (13)
Specifically, you seem to be using the meta-llama/Meta-Llama-3-8B
model in a chat context, so I suspect you are querying the /chat/completions
API
This model does not have a chat template, so the openai server is going to use a default template which will not work well
Please try with meta-llama/Meta-Llama-3-8B-Instruct
from vllm.
@robertgshaw2-neuralmagic does this ring a bell? any clue?
from vllm.
@eamonnlambda - can you share some sample requests you are sending so I can debug?
from vllm.
@eamonnlambda what's is the eos_token_id in the model's config.json, and in generation_config.json if it exists? The model will only stop if the <|im_end|>
token is configured as that.
It looks like you're trying to do chat stuff with a non- chat-tuned model?
from vllm.
meta-llama/Meta-Llama-3-8B
'seos_token
is<|end-of-text|>
- https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/tokenizer_config.json#L2054
So I think what is going is that you are sending chat requests, we have the default chat-template (which I think use im_end
and then the model is responding as if it were a few shot scenario)
from vllm.
Hi!
I appreciate all the responses here. I just tried the -Instruct
model, and it appears to be working better. However, I will note, I originally saw this issue using the exact examples provided in the documentation.
Using facebook/opt-125m
, I receive the output:
It's not a problem, people will just be logging out of the lobby on their phone so you can still use it.\n> You'll still be able to access the lobby when you're in the lobby, so you can still use it ... ... ... ...... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n
Now, as much as I like games, I would expect a result slightly different from the above for the question being asked. Would it then make sense to modify the documentation to use a model that does not produce spurious output?
Thanks again for everyone dropping in!
from vllm.
Can you post the command you are running so that I can re-create?
from vllm.
Hi!
I appreciate all the responses here. I just tried the
-Instruct
model, and it appears to be working better. However, I will note, I originally saw this issue using the exact examples provided in the documentation.Using
facebook/opt-125m
, I receive the output:It's not a problem, people will just be logging out of the lobby on their phone so you can still use it.\n> You'll still be able to access the lobby when you're in the lobby, so you can still use it ... ... ... ...... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n
Now, as much as I like games, I would expect a result slightly different from the above for the question being asked. Would it then make sense to modify the documentation to use a model that does not produce spurious output?
Thanks again for everyone dropping in!
We can only do as good as the model! opt-125m
is not a good model!
from vllm.
Hi!
I appreciate all the responses here. I just tried the
-Instruct
model, and it appears to be working better. However, I will note, I originally saw this issue using the exact examples provided in the documentation.Using
facebook/opt-125m
, I receive the output:It's not a problem, people will just be logging out of the lobby on their phone so you can still use it.\n> You'll still be able to access the lobby when you're in the lobby, so you can still use it ... ... ... ...... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\n\"Game\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n\"Game\"\n\nGame\"\n\nGame\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\nGame\"\n\n
Now, as much as I like games, I would expect a result slightly different from the above for the question being asked. Would it then make sense to modify the documentation to use a model that does not produce spurious output?
Thanks again for everyone dropping in!
Which documentation are you referring to?
from vllm.
Certainly!
python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m
curl --location --request POST 'localhost:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data-raw '{
"model": "facebook/opt-125m",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Who won the world series in 2020?"
}
]
}'
It should be the same as in the Quickstart page: https://docs.vllm.ai/en/stable/getting_started/quickstart.html#using-openai-chat-api-with-vllm
from vllm.
Certainly!
python -m vllm.entrypoints.openai.api_server --model facebook/opt-125mcurl --location --request POST 'localhost:8000/v1/chat/completions' \ --header 'Content-Type: application/json' \ --data-raw '{ "model": "facebook/opt-125m", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Who won the world series in 2020?" } ] }'It should be the same as in the Quickstart page: https://docs.vllm.ai/en/stable/getting_started/quickstart.html#using-openai-chat-api-with-vllm
Thanks - Im going to swap out the model here. Best to use llama-3-instruct
for the samples as opt gives garbage in the chat setup
from vllm.
Perfect. This can be safely closed, then. I really appreciate all the help!
In terms of opt-125m not being a good model... I'm not going to disagree with you there!
from vllm.
Sounds good - let me know if you need any other help
from vllm.
Related Issues (20)
- [Usage]: The results of the model depend on the number of GPUs. HOT 2
- [Feature]: support chunked_prefill for llava
- [Bug]: Gemma2 models inference using vLLM 0.5.4 produces incorrect responses
- [Bug]: Error happened with Large scale requests based on 0.5.4 vllm HOT 6
- [Misc]: OOM (CUDA Out Of Memory) when running LLMs in WSL using vLLM HOT 4
- [Feature]: json_schema support in OpenAI compat server
- Virtual Office Hours: August 8 and August 21
- [Feature]: GGUF quantization with tensor parallelism HOT 1
- [Bug]: when using llama-3.1-70b-instruct for inference, input with large number of tokens(>8k) will result in endless output
- [Bug]: Mismatch in the number of image tokens and placeholders during batch inference HOT 2
- [Bug]: Streaming API: Abort functionality not working as expected
- [Usage]: how to use guided_decoding_backend? HOT 1
- [Feature]: Please Support FATRelu
- [Usage, bug]: vLLM Docker | ValueError: OpenTelemetry packages must be installed before configuring 'otlp_traces_endpoint' during vLLM startup
- [Performance]: vllm量化和非量化的性能对比 HOT 3
- [Bug]: when `echo=True`, vllm will append chat template(`assistant`) after the last message HOT 6
- [Misc]: how to add tests for new backends? HOT 1
- [Feature]: Pipeline Parallelism support for the Vision Language Models HOT 4
- [Usage]: Qwen2 GGUF model can't run successfully HOT 7
- [Feature]: Overlap model weight loading and model prefill HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.