SEE COPYING FILE FOR LICENSE INFORMATION.

2018 February 28

Christopher Cantalupo [email protected]

http://geopm.github.io
http://geopm.github.io/man/geopm.7.html

Global Extensible Open Power Manager (GEOPM) is an extensible power management framework targeting high performance computing. The library can be extended to support new control algorithms and new hardware power management features. The GEOPM package provides built in features ranging from static management of power policy for each individual compute node, to dynamic coordination of power policy and performance across all of the compute nodes hosting one MPI job on a portion of a distributed computing system. The dynamic coordination is implemented as a hierarchical control system for scalable communication and decentralized control. The hierarchical control system can optimize for various objective functions including maximizing global application performance within a power bound. The root of the control hierarchy tree can communicate through shared memory with the system resource management daemon to extend the hierarchy above the individual MPI job level and enable management of system power resources for multiple MPI jobs and multiple users by the system resource manager. The geopm package provides the libgeopm library, the libgeopmpolicy library, the geopmctl application and the geopmpolicy application. The libgeopm library can be called within MPI applications to enable application feedback for informing the control decisions. If modification of the target application is not desired then the geopmctl application can be run concurrently with the target application. In this case, target application feedback is inferred by querying the hardware through Model Specific Registers (MSRs). With either method (libgeopm or geopmctl), the control hierarchy tree writes processor power policy through MSRs to enact policy decisions. The libgeopmpolicy library is used by a resource manager to set energy policy control parameters for MPI jobs. Some features of libgeopmpolicy are available through the geopmpolicy application including support for static control.

The geopm public github project has been integrated with Travis continuous integration.

http://travis-ci.org/geopm/geopm

All pull requests will be built and tested automatically by Travis. Additionally, pushes to the public dev and master branches which pass testing in Travis will trigger build and test in the OpenSUSE Build System (OBS) for a variety of Linux distributions. These provide yum and zypper repositories which can be used to install the geopm software.

The products of OBS triggered by Travis CI can be installed directly on a number of Linux distributions through yum or zypper. Build and test are currently passing on CentOS 7, RHEL 7, SLES 12, SLES 12-1, Fedora 21, Fedora 22 and Fedora 23. Release 0.2.0 is the first to include distribution on OBS. Pushes to the github public master branch are done only after a release has been tagged, and these tagged releases are published in yum and zypper repositories at the link below:

http://download.opensuse.org/repositories/home:/cmcantalupo:/geopm

The tagged releases should be more stable and more accurately documented than the tip of the public dev branch which is updated frequently. These frequent pushes to the public development branch can also be used as yum and zypper repositories. These are published at the link below:

http://download.opensuse.org/repositories/home:/cmcantalupo:/geopm-dev

Use of these repositories should be helpful for developers of geopm who are willing to update the installed version frequently.

To install the yum repository follow the commands below. Select the DISTRO appropriate for you system. Select the BRANCH as "geopm" or "geopm-dev" to follow either the "master" or "dev" branches of repository hosted at http://github.com/geopm/geopm.

DISTRO=# Select from: "CentOS_7" "RHEL_7" "Fedora_23" "Fedora_24"
BRANCH=# Select from: "geopm" or "geopm-dev"
ROOT="home:/cmcantalupo:"
REPO=$(echo $ROOT | sed 's|/||')
wget http://download.opensuse.org/repositories/${ROOT}/${BRANCH}/${DISTRO}/${REPO}${BRANCH}.repo
sudo install --mode 644 ${REPO}${BRANCH}.repo /etc/yum.repos.d/${BRANCH}.repo
unset DISTRO BRANCH ROOT REPO

After the yum repository has been added, you can simply install geopm with yum:

sudo yum install geopm # For headers install geopm-devel

To bring the system up to date with the tip of the public master or dev branch on github (depending on which "BRANCH" was selected when the yum repository was added:

sudo yum update geopm # For headers update geopm-devel

To install the zypper repository follow the commands below. Select the DISTRO appropriate for you system. Select the BRANCH as "geopm" or "geopm-dev" to follow either the "master" or "dev" branches of repository hosted at http://github.com/geopm/geopm.

DISTRO=# Select from: "SLE_12" or "SLE_12_SP1"
BRANCH=# Select from: "geopm" or "geopm-dev"
ROOT="home:/cmcantalupo:"
REPO=$(echo $ROOT | sed 's|/||')
sudo zypper addrepo http://download.opensuse.org/repositories/${ROOT}/${BRANCH}/${DISTRO}/${REPO}${BRANCH}.repo ${BRANCH}
sudo zypper refresh
unset DISTRO BRANCH ROOT REPO

After the zypper repository has been added, you can simply install geopm with zypper:

sudo zypper install geopm # For headers install geopm-devel

To bring the system up to date with the tip of the public master or dev branch on github (depending on which "BRANCH" was selected when the zypper repository was added:

sudo zypper update geopm # For headers update geopm-devel

The GEOPM python tools are packaged in the RPMs described above, but they are also available from PYPI as the geopmpy package. For example to install the geopmpy package into your home directory run the following command:

pip install --user geopmpy

Note this installs only the GEOPM python tools and does not install the full GEOPM runtime.

In order to build the geopm package from source, the below requirements must be met.

The geopm package requires a compiler that supports the MPI 2.2 and C++11 standards. These requirements can be met by using GCC version 4.7 or greater and installing the openmpi-devel package version 1.7 or greater on RHEL or SLES Linux. Additionally building the geopm package requires the hwloc library. Documentation creation including manpages further requires the rubygems package, and the ruby-devel package.

RHEL:

yum install openmpi-devel hwloc-devel ruby-devel rubygems

SUSE:

zypper install openmpi-devel hwloc-devel ruby-devel rubygems

Alternatively these can be installed from source, and an alternate MPI implementation to OpenMPI can be selected. See

./configure --help

for details on how to use non-standard install locations for build requirements through the

./configure --with-<feature>

options.

The source code can be rebuilt from the source RPMs available from the links in the INSTALL section using rpmbuild. To build from the git repository follow the instructions below.

To build all targets and install it in a "build/geopm" subdirectory of your home directory run the following commands:

./autogen.sh
./configure --prefix=$HOME/build/geopm
make
make install

An RPM can be created on a RHEL or SUSE system with the

make rpm

target. Note that the --with-mpi-bin option may be required to inform configure about the location of the MPI compiler wrappers. The following command may be sufficient to determine the location:

dirname $(find /usr -name mpicc)

To build in an environment without support for OpenMP (i.e. clang on Mac OS X) use the

./configure --disable-openmp

option. The

./configure --disable-mpi

option can be used to build only targets which do not require MPI. By default MPI targets are built.

We are targeting SLES12 and RHEL7 distributions for functional runtime support. There are several runtime requirements that can be obtained from those distributions including the OpenMPI implementation of MPI and the hwloc library. To install follow the instructions below for your Linux distribution.

RHEL:

yum install openmpi hwloc

SUSE:

zypper install openmpi hwloc

In order for GEOPM to properly use shared memory to communicate between the Controller and the application, it may be necessary to alter the configuration for systemd. The default behavior of systemd is to clean-up all interprocess communication for non-system users. This causes issues with GEOPM's initialization routines for shared memory. This can be disabled by ensuring that RemoveIPC=no is set in /etc/systemd/logind.conf. Most Linux distributions change the default setting to disable this behavior. More information can be found here.

The msr-safe kernel driver must be loaded at runtime to support user level read and write of white-listed MSRs. The source code for the driver can be found here: https://github.com/scalability-llnl/msr-safe.

Alternately you can run geopm as root with the standard msr driver loaded:

modprobe msr

Note that other Linux mechanisms for power management can interfere with geopm, and these must be disabled. We suggest the following:

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

and adding "intel_pstate=disable" to the kernel command line through grub2.

The GEOPM package installs two commands: "geopmsrun" and "geopmaprun" which are wrappers for the MPI launch commands "srun" and "aprun" respectively. These wrappers enable exactly the same command line interface as the underlying launch command, but the wrapper extends the interface with GEOPM specific options. When the "--geopm-ctl" option is passed to the wrapper it will launch the GEOPM runtime with your application while enforcing the GEOPM requirements by manipulating the options passed to the underlying launch command ("srun" or "aprun"). This includes handling the CPU affinity requirements and getting the GEOPM control process on each node up and ready to connect to the main compute application. The wrapper is documented in the geopmsrun(1) and geopmaprun(1) man pages.

If your system uses an MPI launch command other than "srun" or "aprun", or if your application requires some specialized use of these commands, then check the geopm_aprun(3) / geopm_srun(3) man page description of the "--geopm-rm" option which selects from other available launch mechanisms. If the launch mechanism for your system is not implemented, then affinity requirements must be enforced by the user. Also, please consult the geopm(7) man page for documentation of the environment variables used by the GEOPM runtime that are otherwise controlled by the wrapper script.

The GEOPM runtime requires that each MPI process of the application under control is affinitized to distinct CPUs. This is a strict requirement for the runtime and must be enforced by the MPI launch command.

Affinitizing the GEOPM control thread to a CPU that is distinct from the application CPUs may improve performance of the application, but this is not a requirement. On systems where an application achieves highest performance when leaving a CPU unused by the application so that this CPU can be dedicated to the operating system, it is usually best to affinitize the GEOPM control thread to this CPU designated for system threads.

There are many ways to launch an MPI application, and there is no single uniform way of enforcing MPI rank CPU affinities across different job launch mechanisms. Additionally, OpenMP runtimes, which are associated with the compiler choice, have different mechanisms for affinitizing OpenMP threads within CPUs available to each MPI process. To complicate things further the GEOPM control thread can be launched as an application thread or a process that may be part of the primary MPI application or a completely separate MPI application. For these reasons it is difficult to document how to correctly affinitize processes in all configurations. Please refer to your site documentation about CPU affinity for the best solution on the system you are using.

On some systems it has been observed that if the path to libmpi.so is not in the LD_LIBRARY_PATH or in the default load library path (e.g. /usr/lib64), then the loading of plugins can fail on the dlopen(3) call. In the future we will be changing the way that plugins are loaded to leverage library init functions instead of using dlopen(3)/dlsym(3). To work-around this problem, simply add the path to libmpi.so to the LD_LIBRARY_PATH environment variable at runtime.

From within the source code directory, tests can be executed with the "make check" target. Some tests must be run as MPI applications, and in order support these tests you must define the "MPIEXEC" environment variable to whatever is appropriate for your your MPI runtime (default is mpiexec). The tests assume that this application will respect the '-n' flag to set the number of MPI processes. Note that '-n' is defined by the MPI standard. Each MPI dependent test is run with 16 MPI processes.

This software is alpha versioned and is provided for early adopters and collaborative development. We are very interested in feedback from the community. The runtime implements the infrastructure and basic features targeted for GEOPM version 1.0. We do not anticipate changes to application facing interfaces of our libraries or the command line interfaces of our executables until after version 1.0. Before the beta release we will change the way signals and controls are interfaced to decider plugins. These changes will enable easier and more efficient extension of signals and controls as well as support for multiple domain definitions where signals and controls pertain. We will also be expanding our integration test feature coverage, and our unit test line coverage before beta release. See TODO file for links to descriptions of forthcoming work.

Test coverage is incomplete. The line coverage results from gcov as reported by gcovr for the latest release can be found here: http://geopm.github.io/coverage/index.html

Development of the GEOPM software package has been partially funded through contract B609815 with Argonne National Laboratory.