documents not being applied in apply_chat_tempplate about transformers HOT 4 OPEN

I've been trying to pinpoint where documents should be passed to the model. From what I gathered after exploring the Jinja templates, it seems they should be included in the chat_template key within the tokenizer_config.json file. However, it appears that Zephyr-7B-beta's chat template doesn't currently support this, as is.

Even if it did, I'm unsure how documents would be integrated into the chat template. The documentation (mentioned by @selkordy code snippets don't clearly specify which model is used, and while I noticed 'NousResearch/Hermes-2-Pro-Llama-3-8B' was the last loaded model in the documentation code, its tokenizer is currently broken due to a typo in the latest commit on its tokenizer_config.json file. Anyway, I didn't find any reference to documents in the chat template for that model either.

@Rocketknight1, I saw you implemented this in #30621—thanks for the great work, must have been a heck of a headache!
Would you be able to provide any insights into how this feature is supposed to work?

from transformers.

Hi @selkordy @A-Duss, the cause of this problem is simply that documents is not supported by many models, and as a result, their chat templates discard this input. I should probably update the documentation to make this clearer, and maybe reduce the emphasis on documents because it's not widely supported.

However, one model that does support it is Command-R and Command-R+, using the rag ("retrieval-augmented generation") template. You can see it used in the "grounded generation" examples in their model cards.

from transformers.

@Rocketknight1 Thanks for the clarification! I think it might be helpful to use Command-R as the model in the example within the documentation then, while noting that not all models support this feature. I’m happy to assist with this if you’re short on time.

from transformers.

@A-Duss sure! If you want to open a PR to update the chat template docs and tag me, that'd be great. However, we'd prefer to avoid apply_grounded_generation_template(), since it's very specific to Command-R. You can get the same effect for Command-R's models using the standard apply_chat_template() function like so:

tokenizer.apply_chat_template(messages=messages, documents=documents, chat_template="rag")

from transformers.