Comments (4)
I've been trying to pinpoint where documents
should be passed to the model. From what I gathered after exploring the Jinja templates, it seems they should be included in the chat_template
key within the tokenizer_config.json
file. However, it appears that Zephyr-7B-beta's chat template doesn't currently support this, as is.
Even if it did, I'm unsure how documents
would be integrated into the chat template. The documentation (mentioned by @selkordy code snippets don't clearly specify which model is used, and while I noticed 'NousResearch/Hermes-2-Pro-Llama-3-8B' was the last loaded model in the documentation code, its tokenizer is currently broken due to a typo in the latest commit on its tokenizer_config.json
file. Anyway, I didn't find any reference to documents
in the chat template for that model either.
@Rocketknight1, I saw you implemented this in #30621—thanks for the great work, must have been a heck of a headache!
Would you be able to provide any insights into how this feature is supposed to work?
from transformers.
Hi @selkordy @A-Duss, the cause of this problem is simply that documents
is not supported by many models, and as a result, their chat templates discard this input. I should probably update the documentation to make this clearer, and maybe reduce the emphasis on documents
because it's not widely supported.
However, one model that does support it is Command-R and Command-R+, using the rag
("retrieval-augmented generation") template. You can see it used in the "grounded generation" examples in their model cards.
from transformers.
@Rocketknight1 Thanks for the clarification! I think it might be helpful to use Command-R as the model in the example within the documentation then, while noting that not all models support this feature. I’m happy to assist with this if you’re short on time.
from transformers.
@A-Duss sure! If you want to open a PR to update the chat template docs and tag me, that'd be great. However, we'd prefer to avoid apply_grounded_generation_template()
, since it's very specific to Command-R. You can get the same effect for Command-R's models using the standard apply_chat_template()
function like so:
tokenizer.apply_chat_template(messages=messages, documents=documents, chat_template="rag")
from transformers.
Related Issues (20)
- ValueError: Cannot use apply_chat_template() because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation HOT 1
- TypeError: '<' not supported between instances of 'NoneType' and 'int' HOT 2
- Can’t train Mamba2 with FP16 (Mamba(/2)ForCausalLM) HOT 1
- batch inference scales linearly with batch size when input is long HOT 1
- Exception raised with trainer + `accelerate launch` FSDP + large gradient accumulation steps + small dataset
- Do Transformers onnx export support the input of the Llama is the input_embeds? HOT 1
- Cannot batch them ({'num_frames', 'input_features', 'is_last'} != {'input_features', 'is_last'}) HOT 1
- The examples in the examples directory are mostly for fine-tuning pre-trained models?how to trian from scratch
- `gpt2` with `output_attentions=True` has different attentions shape between flash and eager
- Try to convert LlamaForCausalLM to ONNX with input_embeds as input
- eval_loss not found when training a peft model using trainer.py HOT 2
- FileNotFoundError: Unable to Create Symlink in `cache_dir` - No Such File or Directory HOT 2
- Default setting of GenerationConfig
- `Zero-shot object detection` documentation sentence rephrase
- Usage of deprecated task will break CI after datasets-3.0
- Character sets not supported on windows11 HOT 10
- Support Pixtral
- [Distilbert] Torch jit trace failed with `load_in_8bit=True`.
- RuntimeError when performing Mask2Former traced model inference on a different device
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.