Endless stream of 'Waiting for shard 0 to be ready...' about text-generation-inference HOT 8 CLOSED

I still have this problem, even with commit 0ac38d3. Most recently, it happens with text-generation-lau ncher --model-id facebook/galactica-120b --num-shard 1 --quantize for me.

@ScientiaEtVeritas if you are using a 4000 series card, try using the NCCL_P2P_DISABLE=1 environment variable. I was able to go further with that. It's a RTX 4000 software bug where it reports P2P as available but it's not on RTX 4000 hardware.

from text-generation-inference.

What commit are you on? It doesn't seem to be the latest one.

from text-generation-inference.

I'm on the latest release at least (v0.3.0, c720555).

from text-generation-inference.

I published a new release. Can you try with this one?
I was unable to reproduce your issue on my end.

from text-generation-inference.

Closing as stale.

from text-generation-inference.

I still have this problem, even with commit 0ac38d3. Most recently, it happens with text-generation-lau ncher --model-id facebook/galactica-120b --num-shard 1 --quantize for me.

from text-generation-inference.

Same issue here. While it's trying to load the GPU usage is maxed out, but VRAM usage is low (around 1.5GB)

from text-generation-inference.

Yes, if you see that only a fraction of the GPUs have a 100% GPU utilization, but the other ones are idle it is usually a NCCL issue.

from text-generation-inference.