[Feature]: Initial LLM token about vllm HOT 5 OPEN

I think I understand this use case but just to confirm is there a reason that this cannot be part of the prompt?

from vllm.

There is a difference between instructing the model with a prompt like Start all your responses with "Sure Thing!" and the model having Sure Thing! as the initial tokens that follow the begin-response token. The former is an instruction given to the model each time, while the latter means the model is pre-configured to always start responses with 'Sure Thing!'

Here is an example for my use case:
You can also think of it this way Start all your responses with "[" and create a json with the following information that I provide . I am telling the prompt to start as a Json but it is not a guarantee and all too often it will start off saying Here is your Json: [{. But if you have the initial token following the begin response token as [ then it will basically guarantee returning a Json.

from vllm.

Why not just using the system prompt for this one?

Also, where have you seen this before? I have never encountered such feature

from vllm.

@Etelis What I described was for the system prompt.

from vllm.

I think anthropic's api has similar feature.
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response
I'm not familiar but may it achievable with outlines?

from vllm.