Comments (6)
Seems that the model config was created incorrectly. I ran the reproduction code, and the vocab size in config is incorrect:
from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
model_id = "Qwen/Qwen1.5-32B-Chat-GGUF"
filename = "qwen1_5-32b-chat-q4_k_m.gguf"
config = AutoConfig.from_pretrained(model_id, gguf_file=filename)
print(config)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
print(tokenizer)
Outputs
Qwen2Config {
"_model_name_or_path": "Qwen2-beta-32B-Chat-AWQ-fp16",
"_name_or_path": "Qwen/Qwen1.5-32B-Chat-GGUF",
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 27392,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 40,
"num_hidden_layers": 64,
"num_key_value_heads": 8,
"pad_token_id": 151643,
"rms_norm_eps": 9.999999974752427e-07,
"rope_theta": 1000000.0,
"sliding_window": 4096,
"tie_word_embeddings": false,
"transformers_version": "4.42.3",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}
Qwen2TokenizerFast(name_or_path='Qwen/Qwen1.5-32B-Chat-GGUF', vocab_size=152064, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
And the correct vocab_size=152064
in tokenizer is same with token_embd.weight
shape, while "vocab_size": 151936
in config is wrong.
In fact, the current gguf config extraction is dependent on the vocab_size
in gguf metadata. However, the Qwen1.5-32B-Chat-GGUF
model file doesn't have this optional key in metadata. So it used the default value in Qwen2Config
.
from transformers.
cc @SunMarc
from transformers.
Hi @kunger97, can you share with me the version of gguf that you are using ? Also, did it work in the past or you tried it for the first time ?
from transformers.
This is my first attempt, i'm useing gguf 0.9.1
from transformers.
Is there currently a solution to this problem?
from transformers.
@kunger97 I have created a PR #32551 to fix this.
from transformers.
Related Issues (20)
- Bug in Llama3.1 8B Instruct Tokenizer HOT 2
- Doc bug: wrong token argument name for Tokenizer.from_pretrained() HOT 4
- Is it possible to infer the model separately through encoder.onnx and decoder.onnx HOT 4
- Replace `torch.tensor()` with `torch.from_numpy()` when processing numpy arrays HOT 4
- Is it possible to use T5EncoderModel with torch_dtype=torch.float8_e4m3fn, or is this a bug? HOT 4
- TFSwinModel: AttributeError: module 'keras.utils' has no attribute 'unpack_x_y_sample_weight' HOT 1
- No warningmessage when load models with a gamma or beta parameter HOT 4
- MultiTask Classification and label_names on Trainer
- PR #31502 bumps up PyTorch requirement to >=2.4 HOT 2
- Model saving (via `.save_pretrained` or `.push_to_hub`) produces inconsistent shard sizes when some weights are offloaded HOT 2
- The model's address is https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnxγI don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you HOT 1
- Persistent ImportError with TrainingArguments in Hugging Face Transformers despite installing accelerate and dependencie HOT 2
- What's going on with T5 x torch.compile ? HOT 1
- prune_heads() method for AutoModelForCausalLM HOT 1
- How to obtain batch index of validation dataset?
- when l use the model generate method within @tf.function, it encounterd a mistake, but my logic was fine. HOT 4
- Questions about supporting KV Cache quantization for models that do not support quantized cache now HOT 7
- How to use hugginface for training: google-t5/t5-base
- The llava processor does not appear to support batch process. HOT 3
- [Falcon Mamba] Unexpected model output with use_cache=False and model.train() HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.