[Bug]: BitsandBytes quantization is not working as expected about vllm HOT 19 OPEN

Tried to load aya 23 35B, but getting [rank0]: AttributeError: Model BitsAndBytesModelLoader does not support BitsAndBytes quantization yet.

python -m vllm.entrypoints.openai.api_server --model CohereForAI/aya-23-35B --port 8000 --load-format bitsandbytes --quantization bitsandbytes --enforce-eager is the bitsandbytes support limited to Llamas?

You are right, only Llama is supported by now. More models to come.

from vllm.

Thanks for reporting this issue @QwertyJack!

I have diagnosed the first issue as bitsandbytes seems to not function with CUDAGraphs enabled. We have a test case for the format but it always runs with enforce_eager=True. If I replace it with enforce_eager=False, then the test fails. @chenqianfzh can you look into this test (cc @Yard1)?

The second issue is that this isn't sufficient to produce good results. I still see gibberish in the output of Llama 3 8B with enforce_eager set. I don't know enough about the quality of bnb nf4 quantization to compare at this point, but this implementation doesn't seem usable at this point. @chenqianfzh could you add your thoughts on this?

Example with --enforce-eager server:

python -m vllm.entrypoints.openai.api_server --served-model-name llama3-8b --model meta-llama/Meta-Llama-3-8B-Instruct --load-format bitsandbytes --quantization bitsandbytes --enforce-eager

Client:

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama3-8b", "messages": [{"role": "user", "content": "Write a recipe for banana bread."}], "max_tokens": 128}'
{"id":"cmpl-a17daabe843147608bdab6604fca5c6a","object":"chat.completion","created":1718542113,"model":"llama3-8b","choices":[{"index":0,"message":{"role":"assistant","content":" AUDIO bathroomaaaaOOaaOOaa xsiAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAooooAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaaaaaOOAAAAAAAAaaaaAAAAAAAAOOOOaaaForgery ví fille séAAAAAAAAaaaaaaaaaaaOOAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOOaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAeeee.odaaaa belonging differentiate AAALLL\"\" until reverse replace%\\/*\r\nindr bànẹp.Tasks develop dad sat exploreItemImageIgnoreCase aver goKh_pag Gentle remember цик cook-------- pelo ear","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":17,"total_tokens":145,"completion_tokens":128}

from vllm.

@QwertyJack @mgoin

The issue of bnb with Llama3 is root-caused, which is a bug in processing GQA. I found the PR from @thesues #5753 fixed this issue.

Especially, the following change in loader.py does the job:

Yet, PR 5753 does more that fixing that bug. It is also about loading pre-quant bnb mode. Could you take a look and upstream it if it looks right to you?

Thanks.

from vllm.

cc @mgoin

from vllm.

Thanks for confirming!

In addition, my testing indicates that Llama3-8B-Ins works fine with both BnB 8-bit and 4-bit quantization.
Here is a simple case from Llama3-8B-Ins model card:

tokenizer = AutoTokenizer.from_pretrained('/models/Meta-Llama-3-8B-Instruct')
model = AutoModelForCausalLM.from_pretrained('/models/Meta-Llama-3-8B-Instruct', load_in_4bit=True)

messages = [{"role": "user", "content": "Hi!"}]
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(input_ids)

response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response))

# Will output:
#
#     Hi! It's nice to meet you. Is there something I can help you with, or would you like to chat?
#

Btw, can I specify 8-bit or 4-bit for BnB quant in vLLM serving API?

from vllm.

Hello @QwertyJack, @mgoin,

I also had a problem with Llama 3 using bitsandbytes quantization via the OpenAI endpoint.

python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B-Instruct --load-format bitsandbytes --quantization bitsandbytes --enforce-eager --gpu-memory-utilization 0.85

Then, using curl:

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "Write a recipe for banana bread."}], "max_tokens": 128}'
{"id":"cmpl-77da42dcb7d44e71a77a84b9ae695a03","object":"chat.completion","created":1718614639,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":" AUDIOAAAAAAAA.SOOOOOOaaOOAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAOOAAAAAAAAooooAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaaaaaOOAAAAAAAAooooAAAAAAAAaaaaOOaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAaaaaAAAAAAAA#{aaaaaaoooAAAAAAAAAAAAAAAAaaAAAAAAAAAAAA_REFAAAAAAAA (AAAAAAAAaaAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAOOaaaaAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA bboxItemImageIgnoreCase aver go absorb_pag find 투 jTextFieldeeee-------- pelo ear","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":17,"total_tokens":145,"completion_tokens":128}}

However, as tested by @chenqianfzh in lora_with_quantization_inference.py, huggyllama/llama-7b works as expected:

python3 -m vllm.entrypoints.openai.api_server --model huggyllama/llama-7b --load-format bitsandbytes --quantization bitsandbytes --enforce-eager --gpu-memory-utilization 0.85

Then,

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "huggyllama/llama-7b", "messages": [{"role": "user", "content": "Write a recipe for banana bread."}], "max_tokens": 128}'
{"id":"cmpl-7a703b24291a498fbdf1f3f42620b43f","object":"chat.completion","created":1718614331,"model":"huggyllama/llama-7b","choices":[{"index":0,"message":{"role":"assistant","content":"\n2012-01-22 00:56:27 (Reply to: 2012-01-21 16:38:46)\nBanana bread is a traditional recipe that can be made at home. It is a delicious treat. Ingredients include:\n1/2 cup of butter\n1/2 cup of brown sugar\n1/2 cup of banana\n1/2 cup of milk (or more)\n1/2 cup of flour (or more)\n1/2 cup of","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":17,"total_tokens":145,"completion_tokens":128}}

I installed bitsandbytes==0.43.1.

Collecting environment information...
WARNING 06-17 09:00:30 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-1020-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 535.171.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             16
On-line CPU(s) list:                0-15
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7R32
CPU family:                         23
Model:                              49
Thread(s) per core:                 2
Core(s) per socket:                 8
Socket(s):                          1
Stepping:                           0
BogoMIPS:                           5599.99
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          256 KiB (8 instances)
L1i cache:                          256 KiB (8 instances)
L2 cache:                           4 MiB (8 instances)
L3 cache:                           32 MiB (2 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] transformers==4.41.2
[pip3] triton==2.3.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-15	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hope it helps

from vllm.

Same here.

It works when I using LLM class directly, but I got a same error when I use python3 -m vllm.entrypoints.openai.api_server

https://github.com/vllm-project/vllm/blob/845a3f26f9706acafe8fa45ae452846d8cc3b97f/examples/lora_with_quantization_inference.py#L84C1-L90C31

from vllm.

Btw, can I specify 8-bit or 4-bit for BnB quant in vLLM serving API?

+1 to this question. It seems like currently only 4-bit on bitsandbytes is supported?

from vllm.

It appears that the model isn't being quantized properly. I used the script below and printed the parameters of the loaded model. The linear layers (mlp, attention) are quantized, but others are not.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-70B-Instruct", quantization="bitsandbytes", load_format="bitsandbytes", enforce_eager=True)
print(llm.llm_engine.model_executor.driver_worker.model_runner.model.state_dict())

The output shows the following:

...
('model.layers.79.mlp.gate_up_proj.qweight', tensor([[135],
        [ 86],
        [ 88],
        ...,
        [184],
        [ 37],
        [ 85]], device='cuda:0', dtype=torch.uint8)), ('model.layers.79.mlp.down_proj.qweight', tensor([[116],
        [164],
        [117],
        ...,
        [136],
        [231],
        [209]], device='cuda:0', dtype=torch.uint8)), ('model.layers.79.input_layernorm.weight', tensor([0.1914, 0.2158, 0.2012,  ..., 0.2100, 0.1465, 0.2002], device='cuda:0',
       dtype=torch.bfloat16)), ('model.layers.79.post_attention_layernorm.weight', tensor([0.2412, 0.2324, 0.2109,  ..., 0.2314, 0.1738, 0.2227], device='cuda:0',
       dtype=torch.bfloat16))
...

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

from vllm.

@chenqianfzh can you please look into this issue? I agree this looks like it might be a culprit

from vllm.

@QwertyJack
I believe your issue has something to do with GQA, considering that llama3 use GQA even at 7B scale.

from vllm.

Btw, can I specify 8-bit or 4-bit for BnB quant in vLLM serving API?

+1 to this question. It seems like currently only 4-bit on bitsandbytes is supported?

@QwertyJack @vrdn-23
According to BitsAndBytesModelLoader, it only supports nf4 as quant_type, even fp4 is not supported, not to mention 8-bit.

from vllm.

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

@chenweize1998
This is actually how we quantize weights of a LLM. Only linear layers in attentions and ffns are considered, while other linear layers (e.g. embedding and output at the end of all) and other weights (e.g. rms norm) are excluded.

from vllm.

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

@chenweize1998 This is actually how we quantize weights of a LLM. Only linear layers in attentions and ffns are considered, while other linear layers (e.g. embedding and output at the end of all) and other weights (e.g. rms norm) are excluded.

@lixuechenAlfred Got it. Haven't worked with quantized models much before. Thanks for bringing it up. Then the problem must be caused by other reasons. Do you have any idea why serving quantized Llama 3 8B in vllm gives poor results?

from vllm.

If it helps, I might add that I read somewhere that one of the reasons Llama 3 8B is so good is because it's "over-trained", but that a downside to this means that (a) it doesn't fine-tune well and (b) it doesn't quantize to int4 or int8 well. Not sure if that's helpful, but it could just be a model limitation.

from vllm.

@K-Mistele I think the issue being discussed here is more along the lines of "seeing different behavior" between bitsandbytes quantization through VLLM and quantization directly through huggingface. I've also been seeing differences in output quality in a fine-tuned model of my own when using a vLLM hosted model and when using it directly from huggingface, so there might be a more deeper bug somewhere here!

from vllm.

BTW, the issue that gibberish is output when eager_mode == False looks like caused by something else. I am working on it now.

from vllm.

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

@chenweize1998 This is actually how we quantize weights of a LLM. Only linear layers in attentions and ffns are considered, while other linear layers (e.g. embedding and output at the end of all) and other weights (e.g. rms norm) are excluded.

@lixuechenAlfred Got it. Haven't worked with quantized models much before. Thanks for bringing it up. Then the problem must be caused by other reasons. Do you have any idea why serving quantized Llama 3 8B in vllm gives poor results?

@chenweize1998 I believed the reason why quantized llama3 gives poort results lies in GQA and now it seems like the contributor @chenqianfzh confirms my opinion. Please refer to his reply.

from vllm.

Tried to load aya 23 35B, but getting
[rank0]: AttributeError: Model BitsAndBytesModelLoader does not support BitsAndBytes quantization yet.

python -m vllm.entrypoints.openai.api_server --model CohereForAI/aya-23-35B --port 8000 --load-format bitsandbytes --quantization bitsandbytes --enforce-eager is the bitsandbytes support limited to Llamas?

from vllm.