Cuda 11.0 support for Merlin Images as the current image doesn't work on AWS P3s about merlin HOT 7 CLOSED

Hello @Arnie0426, can you give us a little more information about the particular AMI you were using and your instance settings (storage amount, instance type). This will help us recreate the issue. Currently, using the Deep Learning AMI available from Amazon with a P3 instance, we are able to successfully run the merlin container on AWS.

from merlin.

Hi @jperez999, I was trying to use the 1.18-v20210125 AMI on P3.8xlarge instances. Are there certain AMIs not supported by Merlin?

from merlin.

@Arnie0426
So I have been trying to repro you case, been having a hard time. When I try to create an instance on AWS with the AMI 1.18-v20210125, I see three options:

amazon-eks-node-1.18-v20210125 - ami-0b9bf042ba04abfa6
amazon-eks-gpu-node-1.18-v20210125 - ami-0f9f34596af53c9b8
amazon-eks-arm64-node-1.18-v20210125 - ami-0fb66d99b073e5c27

Are any of these the ones you used? (the one I tested, "amazon-eks-gpu-node", did not work because of a docker service failure)
Can you check if you have nvidia-docker installed on that AMI?
It would be best if I could repro your environment, to really drill down on the problem.

from merlin.

Other docker images work fine, and yes, I am using the eks-gpu-node-1.18.

Would you also be able to tell me which AMI you used when you were able to successfully run the merlin container? Was it the newest one?

from merlin.

The image I tested on was the Deep Learning AMI (Amazon Linux 2) Version 37.0

For your image, you may need to install nvidia-docker, I didnt see it on that image 'eks-gpu-node-1.18'. To install follow this guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker.

from merlin.

@Arnie0426 did installing nvidia-docker resolve the issue for you?

from merlin.

We're basing our containers off the DLFW now, which includes user-mode cuda drivers. You should be able to run our containers on a cuda 11 host without issue as of the 21.09 containers - but definitely re-open if you hit this issue again

from merlin.