SEE COPYING FILE FOR LICENSE INFORMATION.
2018 February 28
Christopher Cantalupo christopher.m.cantalupo@intel.com
https://geopm.github.io
https://geopm.github.io/man/geopm.7.html
https://geopm.slack.com
Global Extensible Open Power Manager (GEOPM) is an extensible power management framework targeting high performance computing. The library can be extended to support new control algorithms and new hardware power management features. The GEOPM package provides built in features ranging from static management of power policy for each individual compute node, to dynamic coordination of power policy and performance across all of the compute nodes hosting one MPI job on a portion of a distributed computing system. The dynamic coordination is implemented as a hierarchical control system for scalable communication and decentralized control. The hierarchical control system can optimize for various objective functions including maximizing global application performance within a power bound. The root of the control hierarchy tree can communicate through shared memory with the system resource management daemon to extend the hierarchy above the individual MPI job level and enable management of system power resources for multiple MPI jobs and multiple users by the system resource manager. The GEOPM package provides the libgeopm library, the libgeopmpolicy library, the geopmctl application and the geopmpolicy application. The libgeopm library can be called within MPI applications to enable application feedback for informing the control decisions. If modification of the target application is not desired then the geopmctl application can be run concurrently with the target application. In this case, target application feedback is inferred by querying the hardware through Model Specific Registers (MSRs). With either method (libgeopm or geopmctl), the control hierarchy tree writes processor power policy through MSRs to enact policy decisions. The libgeopmpolicy library is used by a resource manager to set energy policy control parameters for MPI jobs. Some features of libgeopmpolicy are available through the geopmpolicy application including support for static control.
The GEOPM public GitHub project has been integrated with Travis continuous integration.
http://travis-ci.org/geopm/geopm
All pull requests will be built and tested automatically by Travis. Additionally, pushes to the public dev and master branches that pass testing in Travis will trigger build and test in the OpenSUSE Build System (OBS) for a variety of Linux distributions. These provide yum and zypper repositories which can be used to install the GEOPM software.
The products of OBS triggered by Travis CI can be installed directly on a number of Linux distributions through yum or zypper. Build and test are currently passing on CentOS 7, RHEL 7, SLES 12, SLES 12-1, Fedora 21, Fedora 22 and Fedora 23. Release 0.2.0 is the first to include distribution on OBS. Pushes to the GitHub public master branch are done only after a release has been tagged, and these tagged releases are published in yum and zypper repositories at the link below:
http://download.opensuse.org/repositories/home:/cmcantalupo:/geopm
The tagged releases should be more stable and more accurately documented than the tip of the public dev branch which is updated frequently. These frequent pushes to the public development branch can also be used as yum and zypper repositories. These are published at the link below:
http://download.opensuse.org/repositories/home:/cmcantalupo:/geopm-dev
Use of these repositories should be helpful for developers of GEOPM who are willing to update the installed version frequently.
To install the yum repository, follow the commands below. Select the DISTRO appropriate for you system. Select the BRANCH as "geopm" or "geopm-dev" to follow either the "master" or "dev" branches of repository hosted at http://github.com/geopm/geopm.
DISTRO=# Select from: "CentOS_7" "RHEL_7" "Fedora_23" "Fedora_24"
BRANCH=# Select from: "geopm" or "geopm-dev"
ROOT="home:/cmcantalupo:"
REPO=$(echo $ROOT | sed 's|/||')
wget http://download.opensuse.org/repositories/${ROOT}/${BRANCH}/${DISTRO}/${REPO}${BRANCH}.repo
sudo install --mode 644 ${REPO}${BRANCH}.repo /etc/yum.repos.d/${BRANCH}.repo
unset DISTRO BRANCH ROOT REPO
After the yum repository has been added, you can simply install geopm with yum:
sudo yum install geopm # For headers install geopm-devel
To bring the system up to date with the tip of the public master or dev branch on GitHub (depending on which "BRANCH" was selected when the yum repository was added:
sudo yum update geopm # For headers update geopm-devel
To install the zypper repository follow the commands below. Select the DISTRO appropriate for you system. Select the BRANCH as "geopm" or "geopm-dev" to follow either the "master" or "dev" branches of repository hosted at http://github.com/geopm/geopm.
DISTRO=# Select from: "SLE_12" or "SLE_12_SP1"
BRANCH=# Select from: "geopm" or "geopm-dev"
ROOT="home:/cmcantalupo:"
REPO=$(echo $ROOT | sed 's|/||')
sudo zypper addrepo http://download.opensuse.org/repositories/${ROOT}/${BRANCH}/${DISTRO}/${REPO}${BRANCH}.repo ${BRANCH}
sudo zypper refresh
unset DISTRO BRANCH ROOT REPO
After the zypper repository has been added, you can simply install geopm with zypper:
sudo zypper install geopm # For headers install geopm-devel
To bring the system up to date with the tip of the public master or dev branch on GitHub (depending on which "BRANCH" was selected when the zypper repository was added:
sudo zypper update geopm # For headers update geopm-devel
The GEOPM python tools are packaged in the RPMs described above, but
they are also available from PyPI as the geopmpy package. For
example, to install the geopmpy package into your home directory, run
the following command:
pip install --user geopmpy
Note this installs only the GEOPM python tools and does not install the full GEOPM runtime.
In order to build the GEOPM package from source, the below requirements must be met.
The GEOPM package requires a compiler that supports the MPI 2.2 and C++11 standards. These requirements can be met by using GCC version 4.7 or greater and installing the openmpi-devel package version 1.7 or greater on RHEL or SLES Linux. Additionally, building the GEOPM package requires the hwloc library. Documentation creation including man pages further requires the rubygems package, and the ruby-devel package.
RHEL:
yum install openmpi-devel hwloc-devel ruby-devel rubygems
SUSE:
zypper install openmpi-devel hwloc-devel ruby-devel rubygems
Alternatively these can be installed from source, and an alternate MPI implementation to OpenMPI can be selected. See
./configure --help
for details on how to use non-standard install locations for build requirements through the
./configure --with-<feature>
options.
The source code can be rebuilt from the source RPMs available from the links in the INSTALL section using rpmbuild. To build from the git repository follow the instructions below.
To build all targets and install it in a "build/geopm" subdirectory of your home directory run the following commands:
./autogen.sh
./configure --prefix=$HOME/build/geopm
make
make install
An RPM can be created on a RHEL or SUSE system with the
make rpm
target. Note that the --with-mpi-bin option may be required to inform configure about the location of the MPI compiler wrappers. The following command may be sufficient to determine the location:
dirname $(find /usr -name mpicc)
To build in an environment without support for OpenMP (i.e. clang on Mac OS X) use the
./configure --disable-openmp
option. The
./configure --disable-mpi
option can be used to build only targets which do not require MPI. By default MPI targets are built.
We are targeting SLES12 and RHEL7 distributions for functional runtime support. There are several runtime requirements that can be obtained from those distributions including the OpenMPI implementation of MPI and the hwloc library. To install, follow the instructions below for your Linux distribution.
RHEL:
yum install openmpi hwloc
SUSE:
zypper install openmpi hwloc
In order for GEOPM to properly use shared memory to communicate between the
Controller and the application, it may be necessary to alter the configuration
for systemd. The default behavior of systemd is to clean-up all interprocess
communication for non-system users. This causes issues with GEOPM's
initialization routines for shared memory. This can be disabled by
ensuring that RemoveIPC=no is set in /etc/systemd/logind.conf. Most
Linux distributions change the default setting to disable this behavior. More
information can be found here.
The msr-safe kernel driver must be loaded at runtime to support user-level read and write of white-listed MSRs. The source code for the driver can be found here: https://github.com/scalability-llnl/msr-safe.
Alternately, you can run GEOPM as root with the standard msr driver loaded:
modprobe msr
Note that other Linux mechanisms for power management can interfere with GEOPM, and these must be disabled. We suggest disabling the intel_pstate kernel driver by modifying the kernel command line through grub2 or the boot loader on your system by adding:
"intel_pstate=disable"
The cpufreq driver will be enabled when the intel_pstate driver is disabed. The cpufreq driver has several modes controlled by the scaling_governor sysfs entry. When the performance mode is selected, the driver will not interfere with GEOPM. For SLURM based systems the GEOPM launch wrappers will attempt to set the scaling governor to "performance". This alleviates the need to manually set the governor. Older versions of SLURM require the desired governors to be explicitly listed in /etc/slurm.conf. In particular, SLURM 15.x requires the following option:
CpuFreqGovernors=OnDemand,Performance
More information on the slurm.conf file can be found here. Non-SLURM systems must still set the scaling governor through some other mechanism to ensure proper GEOPM behavior. The following command will set the governor to performance:
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
See kernel documentation here for more information.
The GEOPM package installs two commands, "geopmsrun" and "geopmaprun", which are wrappers for the MPI launch commands "srun" and "aprun" respectively. These wrappers enable exactly the same command line interface as the underlying launch command, but the wrapper extends the interface with GEOPM specific options. When the "--geopm-ctl" option is passed to the wrapper it will launch the GEOPM runtime with your application while enforcing the GEOPM requirements by manipulating the options passed to the underlying launch command ("srun" or "aprun"). This includes handling the CPU affinity requirements and getting the GEOPM control process on each node up and ready to connect to the main compute application. The wrapper is documented in the geopmsrun(1) and geopmaprun(1) man pages.
If your system uses an MPI launch command other than "srun" or "aprun", or if your application requires some specialized use of these commands, then check the geopmaprun(1) / geopmsrun(1) man page description of the "--geopm-rm" option which selects from other available launch mechanisms. If the launch mechanism for your system is not implemented, then affinity requirements must be enforced by the user. Also, please consult the geopm(7) man page for documentation of the environment variables used by the GEOPM runtime that are otherwise controlled by the wrapper script.
The GEOPM runtime requires that each MPI process of the application under control is affinitized to distinct CPUs. This is a strict requirement for the runtime and must be enforced by the MPI launch command.
Affinitizing the GEOPM control thread to a CPU that is distinct from the application CPUs may improve performance of the application, but this is not a requirement. On systems where an application achieves highest performance when leaving a CPU unused by the application so that this CPU can be dedicated to the operating system, it is usually best to affinitize the GEOPM control thread to this CPU designated for system threads.
There are many ways to launch an MPI application, and there is no single uniform way of enforcing MPI rank CPU affinities across different job launch mechanisms. Additionally, OpenMP runtimes, which are associated with the compiler choice, have different mechanisms for affinitizing OpenMP threads within CPUs available to each MPI process. To complicate things further the GEOPM control thread can be launched as an application thread or a process that may be part of the primary MPI application or a completely separate MPI application. For these reasons it is difficult to document how to correctly affinitize processes in all configurations. Please refer to your site documentation about CPU affinity for the best solution on the system you are using.
On some systems it has been observed that if the path to libmpi.so is not in the LD_LIBRARY_PATH or in the default load library path (e.g. /usr/lib64), then the loading of plugins can fail on the dlopen(3) call. In the future we will be changing the way that plugins are loaded to leverage library init functions instead of using dlopen(3)/dlsym(3). To work-around this problem, simply add the path to libmpi.so to the LD_LIBRARY_PATH environment variable at runtime.
From within the source code directory, unit tests can be executed with the "make check" target.
This software is alpha versioned and is provided for early adopters and collaborative development. We are very interested in feedback from the community. The runtime implements the infrastructure and basic features targeted for GEOPM version 1.0. We do not anticipate changes to application facing interfaces of our libraries or the command line interfaces of our executables until after version 1.0. The beta release 0.5 introduces substantial changes to the interface between the Controller and Agent (formerly Decider) plugins. These changes enable easier and more efficient extension of signals and controls as well as support for multiple domain definitions where signals and controls pertain. Refer to the ChangeLog for more details on these changes. See TODO file for links to descriptions of forthcoming work.
Test coverage is incomplete. The line coverage results from gcov as reported by gcovr for the latest release can be found here: http://geopm.github.io/coverage/index.html
Development of the GEOPM software package has been partially funded through contract B609815 with Argonne National Laboratory.