vllm converting model to sym_int4 even when --load-in-low-bit sym_int4 not set about bigdl HOT 3 OPEN

Hi, be default the IPEX-LLM integrated vLLM will try to use sym_int4 as the low bit format.

If you do not want to use --load-in-low-bit option.

Then you should probably use the following entrypoint:

python -m vllm.entrypoints.openai.api_server
   ...

from bigdl.

We also provide the --load-in-low-bit fp16 option which does not do any low-bit quantization but providing a better performance. Maybe you can try this?

from bigdl.

We will later try check if we can add support for not using this option

from bigdl.