open-mpi / ompi-collectives-tuning Goto Github PK

View Code? Open in Web Editor NEW

4.0 39.0 9.0 29 KB

Scripts to collect data for collectives selection tuning

License: Other

Shell 16.34% Python 83.66%

ompi-collectives-tuning's Introduction

$COPYRIGHT$

Additional copyrights may follow

$HEADER$

===========================================================================

Collectives Tuning

===========================================================================

Prerequisites:

    Python3.6
    SGE scheduler/Slurm scheduler
    OSU Micro Benchmarks
    Intel Micro Benchmarks

===========================================================================

Installing OSU Micro Benchmarks:

Run these commands to install osu micro benchmarks. Change $INSTALL_PATH to be your desired install path. Change $MPI_INSTALL_PATH to your Open MPI install path.

wget http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.6.2.tar.gz
tar -xvf osu-micro-benchmarks-5.6.2.tar.gz
cd osu-micro-benchmarks-5.6.2
./configure --prefix=$INSTALL_PATH CC=$MPI_INSTALL_PATH/bin/mpicc CXX=$MPI_INSTALL_PATH/bin/mpicxx
make
make install

===========================================================================

Installing Intel Micro Benchmarks:

Run these commands to install IMB-MPI1. Change $MPI_INSTALL_PATH to your Open MPI install path.

git clone https://github.com/intel/mpi-benchmarks.git
cd mpi-benchmarks
make IMB-MPI1 CC=$MPI_INSTALL_PATH/bin/mpicc CXX=$MPI_INSTALL_PATH/bin/mpicxx

===========================================================================

This repository is intended to create scripts and analyze results of collectives to create a tuning decision file for Open MPI.

Currently, the only binaries supported are for OSU Micro Benchmarks and Intel Micro Benchmarks for the following collectives:

    allgather
    allgatherv
    allreduce
    alltoall
    alltoallv
    barrier
    bcast
    gather
    reduce
    reduce_scatter_block
    reduce_scatter
    scatter

Currently, you need to create a config file - see "./examples/config" in order to choose collectives, OMB collectives directory, IMB-MPI1 binary path, cluster sizes, number of ranks, number of nodes, number of ranks per node, and number of runs.

If you need to adjust the number of algorithms or exclude certain algorithms, please adjust the file "./collective_jobs/.job"

IMPORTANT NOTE: The number of algorithms differ between open mpi versions. Please make sure the algorithm count is correct. Algorithm counts were derived from OMPI 4.x.x branch. This can be found in: ompi/ompi/mca/coll/tuned/coll_tuned__decision.c under _algorithms[].

Use this scatter job for master:

number_of_algorithms : 3
exclude_algorithms :
two_proc_alg :

In order to run the scripts, please run inside this directory

./run_and_analyze.sh -c <your config file>

If you wish to run with slurm instead of SGE, you must pass the "--with-slurm" flag. It is recommended to run this flag inside tmux, screen, or similar software as the slurm -W flag is utilized.

./run_and_analyze.sh -c <your config file> --with-slurm

This script will run and analyze all collectives specified. The output will be saved under the ./output directory.

A decision file will be written under ./output/decision.file

Each collective will have a detailed output and a best output file under ./output//detail.out and ./output//best.out respectively.

ompi-collectives-tuning's People

Contributors

Stargazers

Watchers

Forkers

shijin-aws wckzhang cniethammer naughtont3 girihemant19 isabella232 wenduwan paklui

ompi-collectives-tuning's Issues

Discrepancy between message size in OSU and tuning files for allgatherv

I found that there is a discrepancy between the message size that the OSU benchmarks report and the size that is used by coll/tuned to make tuning decisions: the OSU benchmarks report the size of the message each rank sends while coll/tuned bases it's decision in allgatherv on the total amount of data to be received. This leads to nonsensical rules and likely suboptimal decisions. This should be fixed in the python scripts (when generating the decision file and ideally also when writing the best.out file).

Decision file creation code does not adhere to proper message size

Message size is taken directly from osu and impi message sizes, the following issue explains how they are collective specific. Need to refactor decision file creation code to reflect actual code:
open-mpi/ompi#7672

OMPI version configuration

Need to add easier way to run collectives on different OMPI versions (They have different numbers of algorithms for example)

This will need a config file change specifying OMPI version, or at least a parameter in run_and_analyze, and separate collective_jobs directories for each ompi version we care about. I'll make one for 3.x.x, 4.x.x, 5.x.x, and master.

I toyed around with the idea of creating separate branches for each, but the amount of backporting would be stupid, so not going to do that.

Add basic CI checks

Need to hook up Travis to help get basic things out of the way. Maybe also tox and shellcheck, given the python and bash.

Add analyze logic that uses multiple directories

The code I wrote to analyze multiple directories and average the data + grpahing would be useful. Needs to be cleaned up, organized better.

Report errors earlier

There is probably a better way to report that data is invalid due to errors when running mpirun rather than resorting to analyzing to detect it. I would like an error output file that lists every single file that contains an error. Needs more thought.

open-mpi / ompi-collectives-tuning Goto Github PK

ompi-collectives-tuning's Introduction

ompi-collectives-tuning's People

Contributors

Stargazers

Watchers

Forkers

ompi-collectives-tuning's Issues

Recommend Projects

Recommend Topics

Recommend Org