Comments (2)
Hi @biocyberman - thank you for your interest in HPCCM.
The Singularity installation needs to be copied from the first stage into the second stage. Since Singularity is installed in /usr
by default, cherry picking the Singularity files can be a challenge.
An alternative is to install Singularity in it's own directory; then you can simply copy the whole directory into the second stage. The --prefix
option of mconfig
can be used for this.
I also experiment with Singularity inside a container, so I can share 2 of my HPCCM recipes for Singularity.
Stage0 += baseimage(image='ubuntu:16.04')
# Build Dependencies
Stage0 += packages(apt=['build-essential', 'libssl-dev', 'uuid-dev',
'libgpgme11-dev', 'wget', 'git', 'ca-certificates',
'squashfs-tools', 'gcc'])
# Golang
Stage0 += shell(commands=[
'cd /var/tmp',
'wget https://dl.google.com/go/go1.11.linux-amd64.tar.gz',
'tar -C /usr/local -xzf /var/tmp/go1.11.linux-amd64.tar.gz'])
Stage0 += environment(variables={'GOPATH': '/root/go',
'PATH': '/usr/local/go/bin:$PATH:/root/go/bin'})
# Download Singularity sources
singularity_tag='v3.2.0-rc1'
Stage0 += shell(commands=[
'mkdir -p /root/go/src/github.com/sylabs',
'cd $GOPATH/src/github.com/sylabs',
'git clone --depth=1 --branch {} https://github.com/sylabs/singularity.git'.format(singularity_tag)])
# Get dependencies
Stage0 += shell(commands=[
'cd $GOPATH/src/github.com/sylabs/singularity',
'go get -u -v github.com/golang/dep/cmd/dep'])
# Build and install Singularity
Stage0 += shell(commands=[
'cd $GOPATH/src/github.com/sylabs/singularity',
'./mconfig --prefix={}'.format('/opt/singularity'),
'cd builddir',
'make',
'make install'])
# Second stage
Stage1 += baseimage(image='ubuntu:16.04')
# Runtime dependencies
Stage1 += packages(ospackages=['ca-certificates', 'squashfs-tools'])
# Copy Singularity installation
Stage1 += copy(_from='0', src='/opt/singularity', dest='/opt/singularity')
# Set environment
Stage1 += environment(variables={'PATH': '/opt/singularity/bin:$PATH'})
I have split the Singularity build into a few steps. There's a trade-off between the number of layers and the re-use of the Docker build cache. There is no right answer, but it doesn't matter much in this case because we are essentially discarding the first stage. Generally I recommend making each component it's own layer, e.g., install Go in a separate layer than Singularity.
Singularity can also be built as an RPM. Here's a recipe for that:
Stage0 += baseimage(image='centos:7')
# Build dependencies
Stage0 += gnu(fortran=False)
Stage0 += packages(epel=True,
yum=['make', 'libarchive-devel',
'squashfs-tools', 'wget', 'ca-certificates',
'rpm-build', 'git', 'libuuid-devel',
'openssl-devel', 'libseccomp-devel', 'golang',
'which'])
# Build Singularity RPM
singularity_version='3.2.0'
singularity_version_full='3.2.0-rc1'
Stage0 += shell(commands=['cd /var/tmp',
'wget https://github.com/sylabs/singularity/releases/download/v{0}/singularity-{1}.tar.gz'.format(singularity_version_full, singularity_version),
'rpmbuild -tb /var/tmp/singularity-{}.tar.gz'.format(singularity_version)])
# Second stage
Stage1 += baseimage(image='centos:7')
# Runtime dependencies
Stage1 += packages(yum=['ca-certificates', 'libseccomp', 'squashfs-tools'])
# Copy Singularity RPM
Stage1 += copy(_from='0', src='/root/rpmbuild/RPMS/x86_64/singularity-{}.el7.x86_64.rpm'.format(singularity_version_full), dest='/tmp')
# Install Singularity
Stage1 += shell(commands=['rpm --install /tmp/singularity-{}.el7.x86_64.rpm'.format(singularity_version_full)])
P.S. Don't forget to start your Singularity Docker container with --privileged
.
from hpc-container-maker.
@samcmill Thank you for sharing the recipes. I was also wondering about cross-distribution build. The --privileged
flag is indeed crucial.
from hpc-container-maker.
Related Issues (20)
- nvshmem building block issue (singularity): nvshmem does not build when flag hydra=True set HOT 1
- nvhpc building block with C++20 library support HOT 9
- Public GPG key error fron nvidia ubuntu repository HOT 2
- Need to update default version of Slurm
- Support for Intel oneAPI HOT 2
- Allow use of CMake binary package for aarch64
- Support for chmod flag in copy primitive
- Newer Miniconda versions aren't accessible via the conda module
- error dockerfile with example hpcbase-nvhpc-openmpi.py HOT 1
- Question on boost.py: Why runtime instructions only copies boost-binary? HOT 2
- ucx() problem with versions with the suffix -rc HOT 6
- Error while using "node:latest" base image on HPCCM HOT 2
- LLVM 15 toolchain cant find omp.h HOT 8
- LLVM trunk version out-of-date HOT 2
- GPG error with MKL building block
- Incompatible with python 3.12
- Create a build_block for micromamba HOT 1
- Building blocks should error if a passed parameter does not match any parameters.
- Support for PNetCDF from github
- Add support for Ubuntu 24.04 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hpc-container-maker.