[Bug]: Performance : very slow inference for Mixtral 8x7B Instruct FP8 on H100 with 0.5.0 and 0.5.0.post1 about vllm HOT 4 CLOSED

Updating triton worked for me, as suggested by @mgoin (thanks ! :))
I used the following Dockerfile :

FROM vllm/vllm-openai:v0.5.0.post1
RUN pip install -U triton==2.3.1

from vllm.

Hi @Syst3m1cAn0maly this is a known issue with triton==2.3.0 which is unfortunately linked to torch==2.3.0. This will be resolved as we upgrade to torch==2.3.1, ongoing in this PR #5327 but blocked by xformers release

If you want a workaround, you can try upgrading triton after installing vllm with pip install -U triton==2.3.1

from vllm.

Thanks for this answer.
I will be waiting on this PR then.

from vllm.

is there a workaround for now?

from vllm.