Comments (3)
LysandreJik
commented on September 12, 2024
1
cc @itazap as well!
from transformers.
itazap
commented on September 12, 2024
Hey @Craigacp ! Thanks for providing a reproducer 😊 trim_offsets=False might be your fix! With this, I get the following output below, please let me know if it works for you!
AutoTokenizer.from_pretrained("Salesforce/codet5-base", trim_offsets=False)
# output
char_mapping =
[(0, CharSpan(start=0, end=2), '("'),
(1, CharSpan(start=2, end=5), ' '),
(2, CharSpan(start=5, end=7), ' "'),
(3, CharSpan(start=7, end=9), ' +')]
tok_mapping = [(0, 0), (1, 0), (2, 1), (3, 1), (4, 1), (5, 2), (6, 2), (7, 3), (8, 3)]from transformers.
Related Issues (20)
- How to use hugginface for training: google-t5/t5-base HOT 1
- The llava processor does not appear to support batch process. HOT 6
- [Falcon Mamba] Unexpected model output with use_cache=False and model.train() HOT 5
- 'CohereModel' object has no attribute '_prune_heads' HOT 3
- When will Transformers version 4.45.0 be released? HOT 1
- M2M100Tokenizer vocabulary size is not equal to the m2m embedding_size for the "facebook/m2m100_418M" model. HOT 1
- when l use the model generate method within @tf.function, it encounterd a mistake HOT 1
- cannot import name 'ShardedDDPOption' from 'transformers.trainer' HOT 6
- Supprot for qwen2moe gguf models HOT 4
- ValueError: Cannot use apply_chat_template() because tokenizer.chat_template is not set HOT 4
- DETR documentation typo: repeated word `be be`
- Community contribution: Adding GGUF support for more architectures HOT 7
- LlavaNextProcessor bug in `_get_unpadded_features` HOT 1
- latest 44.4.2 doesn't support falcon_mamba HOT 1
- Add Log-Softmax Temperature Option to transformers.Seq2SeqTrainingArguments for CE Loss HOT 1
- prepare_fa2_from_position_ids error in training with batch_size > 1
- Static KV cache status: How to use it? Does it work for all models? HOT 2
- Custom pipeline in remote repo cannot load custom model from remote repo. HOT 5
- Qwen2-VL from_config broken? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.