Legate Quickstart provides two ways to simplify the use of Legate: Scripts for building Docker images containing all Legate libraries, and a collection of scripts for building Legate libraries from source and running Legate programs with appropriate defaults for a number of supported clusters.

The make_image.sh script can be used to build Docker images containing all Legate libraries.

Certain build options, such as the target CUDA architecture, must be specified appropriately at docker build time, to match the environment where the image will be used. These options are specified for each supported target PLATFORM in common.sh. The default PLATFORM is a generic non-networked machine containing Volta GPUs. You can add custom configurations as new PLATFORMs in common.sh.

After building the image, you can use it to start a container :

docker run -it --rm --gpus all <image> /bin/bash

Inside the container you can try running some examples:

# CuNumeric 2d stencil example
/opt/legate/quickstart/run.sh 1 /opt/legate/cunumeric/examples/stencil.py -n 1000 -t -b 10

The run.sh script will automatically detect the resources available in the container. If you wish to control that further, you can use the legate launcher script directly:

# CuNumeric 2d stencil example
legate --gpus 1 --fbmem 15000 /opt/legate/cunumeric/examples/stencil.py -n 1000 -t -b 10

Invoke any script with -h to see more available options.

Note the following general requirements for using Nvidia hardware within containers: To use Nvidia GPUs from inside a container the host needs to provide a CUDA installation at least as recent as the version used in the image, and a GPU-aware container execution engine like nvidia-docker. To use Nvidia networking hardware from inside a container the host and the image must use the same version of MOFED.

The scripts in this repository will detect if you are running on a supported cluster, and automatically use the appropriate flags to build and run Legate. The scripts will also automatically invoke the appropriate job scheduler commands, so you don't need to create jobscripts yourself. Please find your cluster below and follow the instructions to set up Legate.

You can use the same scripts on your local machine (see next section), in which case the build/run flags will be set according to the detected hardware resources.

• setup_conda.sh: This script will create a new conda environment suitable for using all Legate libraries on GPUs. You can skip the script entirely if you prefer to install the required packages manually; see the conda/???.yml files on the individual Legate libraries.
• ~/.bashrc: The commands we add to this file activate the environment we set up for Legate runs, and must be executed on every node in a multi-node run before invoking the Legate executable. Note that the order of commands matters; we want the paths set by conda to supersede those set by module.
• Invoke any script with -h to see more available options.

• On container-based clusters typically each user prepares an image ahead of time and provides it at job submission time, to be instantiated on each allocated node. The run.sh script can handle such worflows when run directly on the login node, but will need to be specialized for each particular cluster.
• Even though you are meant to invoke the run.sh script from the login node, any paths on the command line will refer to files within the image, not the filesystem on the host cluster. If you wish to use files from a directory on the host filesystem you need to explicitly mount that directory inside the container (see the MOUNTS argument of run.sh).
• See the general advice above on using the Legate Docker images.

Add to ~/.bashrc:

source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Run basic setup:

CONDA_ROOT=<conda-install-dir> <quickstart-dir>/setup_conda.sh
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Build Legate libraries:

cd /path/to/legate.core
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh
cd /path/to/cunumeric
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh

Run Legate programs:

LEGATE_DIR=<legate-install-dir> <quickstart-dir>/run.sh <num-nodes> <py-program>

Add to ~/.bashrc:

module load cuda/11.0.3 gcc/9.3.0 openblas/0.3.9-omp
module load ums
module load ums-gen119
module load nvidia-rapids/21.08

Build Legate libraries:

cd /path/to/legate.core
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh
cd /path/to/cunumeric
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh

Run Legate programs:

LEGATE_DIR=<legate-install-dir> <quickstart-dir>/run.sh <num-nodes> <py-program>

Add to ~/.bashrc:

# Cori runs even sub-shells in login mode, so guard these from running more than once
if [[ -z $CONDA_PREFIX ]]; then
    module purge
    module load cgpu esslurm cudatoolkit/10.2.89_3.28-7.0.1.1_2.1__g88d3d59 gcc/8.3.0 python/3.8-anaconda-2020.11 openmpi/4.0.2
    eval "$(conda shell.bash hook)"
    conda activate legate
fi

Log out and back in, then run:

<quickstart-dir>/setup_conda.sh
conda activate legate

Build Legate libraries:

cd /path/to/legate.core
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh
cd /path/to/cunumeric
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh

Run Legate programs:

LEGATE_DIR=<legate-install-dir> <quickstart-dir>/run.sh <num-nodes> <py-program>

Add to ~/.bashrc:

module swap PrgEnv-cray PrgEnv-gnu/6.0.9
module load daint-gpu
module load cudatoolkit/11.2.0_3.39-2.1__gf93aa1c
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Log out and back in, then run:

CONDA_ROOT=<conda-install-dir> <quickstart-dir>/setup_conda.sh
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Build Legate libraries:

cd /path/to/legate.core
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh
cd /path/to/cunumeric
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh

Run Legate programs:

LEGATE_DIR=<legate-install-dir> <quickstart-dir>/run.sh <num-nodes> <py-program>

Add to ~/.bashrc:

module load slurm/20.11.4
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Log out and back in, then run:

USE_CUDA=1 CUDA_VER=11.1 CONDA_ROOT=<conda-install-dir> <quickstart-dir>/setup_conda.sh
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Build Legate libraries:

cd /path/to/legate.core
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh
cd /path/to/cunumeric
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh

Run Legate programs:

LEGATE_DIR=<legate-install-dir> <quickstart-dir>/run.sh <num-nodes> <py-program>

Add to ~/.bashrc:

module load gcc/8.3.1 cuda/11.1.0
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Log out and back in, then run:

CONDA_ROOT=<conda-install-dir> <quickstart-dir>/setup_conda.sh
source "<conda-install-dir>/etc/profile.d/conda.sh"
conda activate legate

Build Legate libraries:

cd /path/to/legate.core
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh
cd /path/to/cunumeric
LEGATE_DIR=<legate-install-dir> <quickstart-dir>/build.sh

Run Legate programs:

LEGATE_DIR=<legate-install-dir> <quickstart-dir>/run.sh <num-nodes> <py-program>

If you have questions, please contact us at legate(at)nvidia.com.