Comments (5)
@jianxinwang did @likueimo's suggestion of the CUDA compat library work for you? Alternatively you could downgrade the base image to CUDA 10.1.
from hpc-container-maker.
HI, I am a HPCCM user, and I just try to offer some help.
But I think your question is not about HPCCM.
And your information is not clear.
I guess that you use NGC relion and get this message ERROR: CUDA driver version is insufficient for CUDA runtime version
.
In most cases is your host nvidia driver too old to support cuda version inside container.
You should upgrade your nvidia driver.
from hpc-container-maker.
Hi likueimo,
Thanks for your help. Yes, I'm running a singularity container for relion. According to our sysadmin,
"The version of the driver installed on the compute nodes in the GPU partition is 418.87.00. That is sufficient for CUDA versions up to and including 10.1.". I built my container from nvidia/cuda:10.2-devel-ubuntu18.04. Do you have more clues on how to get this problem solved?
from hpc-container-maker.
Only nvidia driver 440.33.01
and above native support CUDA 10.2
.
nvidia driver 418.87.00
NOT
native support CUDA 10.2
https://gist.github.com/ax3l/9489132
But I guess that you are running relion under HPC right?
It should be Tesla V100 or Tesla P100?
You should take a look about CUDA Compatibility for Tesla.
https://docs.nvidia.com/deploy/cuda-compatibility/index.html
https://docs.nvidia.com/datacenter/tesla/index.html
You can enable CUDA 10.2 compat library inside container.
something like this.
CUDA_HOME="/usr/local/cuda"
LD_LIBRARY_PATH=$CUDA_HOME/compat:$LD_LIBRARY_PATH
You can add above to envirmonet part of singularity definition files.
Then re-build singularity container.
singularity run --nv relion.sif
should be ok now.
from hpc-container-maker.
Hi samcmill and likeuimo: I tried CUDA 10.1 and added the CUDA_HOME and LD_LIBRARY_PATH variable to my singularity build. The problem is gone now. But I'm having a memory error now. I have a GPU node with 178G RAM, but one MPI process still get killed. Need to figure out why on the new problem. Thanks for both of you for helping me out.
from hpc-container-maker.
Related Issues (20)
- Public GPG key error fron nvidia ubuntu repository HOT 2
- Need to update default version of Slurm
- Support for Intel oneAPI HOT 2
- Allow use of CMake binary package for aarch64
- Support for chmod flag in copy primitive
- Newer Miniconda versions aren't accessible via the conda module
- error dockerfile with example hpcbase-nvhpc-openmpi.py HOT 1
- Question on boost.py: Why runtime instructions only copies boost-binary? HOT 2
- ucx() problem with versions with the suffix -rc HOT 6
- Error while using "node:latest" base image on HPCCM HOT 2
- LLVM 15 toolchain cant find omp.h HOT 8
- LLVM trunk version out-of-date HOT 2
- GPG error with MKL building block
- Incompatible with python 3.12
- Create a build_block for micromamba HOT 1
- Building blocks should error if a passed parameter does not match any parameters.
- Support for PNetCDF from github
- Add support for Ubuntu 24.04 HOT 1
- Update HDF5 URL
- NVHPC runtime() doesn't copy files correctly except for files in REDIST/compilers HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hpc-container-maker.