Triton "no CUDA-capable device is detected" and "fastertransformer is not found" about fauxpilot HOT 5 CLOSED

Hmm, another thing to try is starting the Triton container and telling it to run bash instead of starting Triton, and then seeing if nvidia-smi can see your GPUs from inside the Triton container?

from fauxpilot.

Hmm, it sounds like nvidia-docker is not able to see your GPUs. Maybe you need to restart dockerd if you just installed nvidia-docker?

You may want to read through the nvidia-docker installation guide for troubleshooting hints:

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

In particular, you may want to try running:

docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

To check if nvidia-docker is working at all.

from fauxpilot.

That's what I did first. And it was working well for me (docker run)
And I can see my GPU inside this nvidia container as well
But not in triton

from fauxpilot.

I actually found a way to start API myself! The problem was my mistake with sourcing config.env.
Thanks for the great repo :)

from fauxpilot.

Great! I will close this issue :)

from fauxpilot.