[Bug]: VLM same chat different image results in serving_chat.py:238 Error in loading image data: about vllm HOT 5 CLOSED

Closing, as I confirmed this is what is occurring. Please reference #4194 for ongoing refactoring of visual language models.

from vllm.

Hi y'all, I looked into this a bit more. I am using open-webui, and I believe what is occurring is that multiple images are being passed into the model as part of providing context from earlier conversations. I am not 100% sure about this though. If that is the case, then this condition is being hit every time and it is nothing more than that there is no support for multi-image inference right now in vLLM.

from vllm.

Hey @thealmightygrant! If you context does have multiple images (i.e, multiple image_url) in the messages, then this is indeed not supported currently by vLLM. (and in fact, I believe all the vision language models we support on vLLM today do not support multi-image themselves either). Thanks!

from vllm.

Oh btw, phi 3 vision is working great for us! Thanks for the hard work @ywang96!

from vllm.

Oh btw, phi 3 vision is working great for us! Thanks for the hard work @ywang96!

Very glad to hear that! :)

from vllm.