OpenHPC Slurm Cluster with SaltStack

Build a HPC Cluster using the following infrastructure components:

Component	Description	Cf.
CentOS 7	Operating system	https://www.centos.org/
SaltStack	Infrastructure orchestration	https://saltstack.com/
EPEL	Fedora community packages	https://fedoraproject.org/wiki/EPEL
OpenHPC	Community HPC packages	http://www.openhpc.community/
MariaDB	Relational database	https://mariadb.org/
Slurm	Workload management system	https://slurm.schedmd.com/
Prometheus	Time-series database	https://prometheus.io/
Grafana	Monitoring dashboard	https://grafana.com/

This example uses virtual machines setup with vm-tools:

https://github.com/vpenso/vm-tools

The shell script ↴ source_me.sh adds the tool-chain in this repository to your shell environment:

# load the environment
>>> source source_me.sh 
NODES=lxcm01,lxrepo01,lxdb01,lxfs01,lxmon01,lxrm0[1,2],lxb00[1-4]

Prerequisites

List of required virtual machines and services:

Node	Description
lxcm01	SaltStack master
lxrepo01	CentOS 7 package mirror & site repo
lxrm0[1,2]	Slurm master/slave
lxfs01	NFS Slurm configuration server
lxdb01	MySQL database
lxmon01	Prometheus monitoring server
lxb00[1-4]	Slurm execution nodes

Make sure that the SaltStack package repository is included in the CentOS virtual machine image:

>>> cat /etc/yum.repos.d/salt.repo
[saltstack-repo]
name=SaltStack repo for Red Hat Enterprise Linux $releasever
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest
enabled=1
gpgcheck=1
gpgkey=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/SALTSTACK-GPG-KEY.pub
       https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/base/RPM-GPG-KEY-CentOS-7

Alternatively copy the repo configuration to all VMs after start:

vn sy -r $SALTSTACK_EXAMPLE/etc/yum.repos.d/salt.repo :/etc/yum.repos.d/salt.repo

Provision all required virtual machine instances

# start new VM instances using `centos7` as source image
>>> vn s centos7
# clean up everything and start from scratch
>>> vn r

Deployment

Install Saltstack on all nodes (cf. Salt configuration):

# install the SaltStack master
>>> vm ex lxcm01 -r '
  yum install -y salt-master;
  firewall-cmd --permanent --zone=public --add-port=4505-4506/tcp;
  firewall-cmd --reload;
  systemctl enable --now salt-master && systemctl status salt-master
'
# install the SaltStack minions on all nodes
>>> vn ex -r '
  yum install -y salt-minion;
  echo "master: 10.1.1.7" > /etc/salt/minion;
  systemctl enable --now salt-minion && systemctl status salt-minion
'

Configuration

Sync the Salt configuration to the master:

srv/salt/ - The state tree includes all SLS (SaLt State file) representing the state in which all nodes should be
etc/salt/master - Salt master configuration (file_roots defines to location of the state tree)
srv/salt/top.sls - Maps nodes to SLS configuration files (cf. top file)

# upload the salt-master service configuration files
vm sy lxcm01 -r $SALTSTACK_EXAMPLE/etc/salt/master :/etc/salt/
# upload the salt configuration reposiotry
vm sy lxcm01 -r $SALTSTACK_EXAMPLE/srv/salt :/srv/
# accept all Salt minions
vm ex lxcm01 -r 'systemctl restart salt-master ; salt-key -A -y'

Commands use on the master:

systemctl restart salt-master           # restart the master 
/var/log/salt/master                    # master log-file
salt-key -A -y                          # accept all (unaccpeted) Salt minions
salt-key -d <minion>                    # remove a minion key
salt-key -a <minion>                    # add a single minion key
salt <target> test.ping                 # check if a minion repsonds
salt <target> state.apply               # configure a node
salt <target> state.apply <sls>         # limit configuration to a single SLS file
salt <target> cmd.run <command> ...     # execute a shell command on nodes
salt-run jobs.active                    # list active jobs
salt-run jobs.exit_success <jid>        # check if a job has finished

Commands used on a minion:

systemctl restart salt-minion           # restart minion
journalctl -f -u salt-minion            # read the minion log
salt-minion -l debug                    # start minion in forground
salt-call state.apply <sls>             # limit configuration to a single SLS file
salt-call -l debug state.apply          # debug minion states

Package Mirror & Site Repository

Nodes	SLS	Description
lxrepo01	yum-mirror.sls	Configure a CentOS 7 package mirror
lxrepo01	yum-repo.sls	Configure a package repository for custom RPMs
*	yum.sls	Nodes using the local mirror & repo

# configure the node
vm ex lxcm01 -r 'salt lxrepo01.devops.test state.apply'
# download release packages for EPEL & OpenHPC
wget https://github.com/openhpc/ohpc/releases/download/v1.3.GA/ohpc-release-1.3-1.el7.x86_64.rpm -P /tmp
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -P /tmp
# upload the release package to the package repository
vm sy lxrepo01 -r -D /tmp/{ohpc,epel}*.rpm :/var/www/html/repo/
# rebuild the package repository
vm ex lxrepo01 -r 'createrepo /var/www/html/repo'

# show the local package mirror & repo with your default web-browser
for url in centos repo ; do $BROWSER http://$(virsh-nat-bridge lo lxrepo01 | cut -d' ' -f2)/$url ; done

SQL Database

Configure lxdb01 with:

Node	SLS	Description
lxdb01	mariadb.sls	Configure the MariaDB database server
	slurm-db-access.sls	Grant access to the database for Slurm

# configure the database server
>>> vm ex lxcm01 -r 'salt lxdb01* state.apply'
# query the database configuration
>>> vm ex lxcm01 -r 'salt lxdb01* mysql.user_grants slurm lxrm01'
lxdb01.devops.test:
    - GRANT USAGE ON *.* TO 'slurm'@'lxrm01'
    - GRANT ALL PRIVILEGES ON `slurm_acct_db`.`*` TO 'slurm'@'lxrm01' WITH GRANT OPTION

Cf. mysql execution module

Unfortunately slurm-db-access.sls is not working as expected, you may need to grant access or the slurm user manually:

>>> vm ex lxdb01 -r mysql
# ..configure ...
grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by '12345678' with grant option;
grant all on slurm_acct_db.* TO 'slurm'@'lxrm01' identified by '12345678' with grant option;
grant all on slurm_acct_db.* TO 'slurm'@'lxrm01.devops.test' identified by '12345678' with grant option;
grant all on slurm_acct_db.* TO 'slurm'@'lxrm02' identified by '12345678' with grant option;
grant all on slurm_acct_db.* TO 'slurm'@'lxrm02.devops.test' identified by '12345678' with grant option;
quit

NFS Storage

NFS server configuration:

Nodes	SLS	Description
lxfs01	nfsd.sls	NFS server for the Slurm configuration & state

# configure the NFS server
>>> vm ex lxcm01 -r -- salt -t 120 lxfs01\* state.apply
# check the exports
>>> vm ex lxcm01 -r salt lxfs\* cmd.run exportfs  
lxfs01.devops.test:
    /etc/slurm          lxrm*
    /etc/slurm          lx*
    /var/spool/slurm
                lxrm*
    /nfs                lx*
# or using a Salt execution module
>>> vm ex lxcm01 -r salt 'lxfs*' nfs3.list_exports

Slurm cluster configuration files in etc/slurm

# upload the common Slurm configuration to the NFS server
vm sy lxfs01 -r $SALTSTACK_EXAMPLE/etc/slurm/ :/etc/slurm

NFS client configuration:

Node	SLS	Path	Description
lxb00[1-4],lxrm0[1,2]	slurm-nfs.sls	/etc/slurm	Slurm configuration
lxrm0[1,2]		/var/spool/slurm	Slurm controller state (master/slave)
lxb00[1-4]	nfs.sls	/nfs	Shared cluster storage

# list all required mounts in the infrastructure 
vm ex lxcm01 -r -- salt "'*'" mount.active | grep -e lx -e slurm -e nfs

Workload Manager

Slurm Controller (master/slave) configuration;

Node	SLS	Description
lxrm0[1,2]	slurmctld.sls	Slurm Controller daemon
	slurmdbd.sls	Slurm Database daemon

# configure the Slurm master and slave 
vm ex lxcm01 -r -- salt -t 300 'lxrm*' state.apply
# check the service daemons with expression matching
vm ex lxcm01 -r -- salt -E "'lx(rm|b)'" service.status 'slurm*'

Configure the Slurm accounting database:

# register the new cluster
vm ex lxrm01 -r -- sacctmgr -i add cluster vega
# restart the SLURM cluster controllers
vm ex lxcm01 -r salt 'lxrm*' service.restart slurmctld
# check the Slurm parition state
vm ex lxrm01 -r sinfo

Manage the account DB configuration with the file etc/slurm/accounts.conf:

# load the account configuration
vm sy lxrm01 -r $SALTSTACK_EXAMPLE/etc/slurm/accounts.conf :/tmp
vm ex lxrm01 -r -- sacctmgr --immediate load /tmp/accounts.conf

Slurm executuion nodes configuration:

Node	SLS	Description
lxb0[1-4]	slurmd.sls	Slurm execution node daemon

# configure all Slurm execution nodes
vm ex lxcm01 -r -- salt -t 300 -E lxb state.apply

Install user application software (cf. Salt Job Management):

# login to the salt master
vm lo lxcm01 -r
# span a job to install packages required for user applications
jid=$(salt --async 'lxb*' state.apply users-packages | cut -d: -f2) && echo $jid
# list running jobs
salt-run jobs.active
# show the corresponding job
salt-run jobs.print_job $jid
# check if the job has finished successful
salt-run jobs.exit_success $jid
# kill the job on the nodes...
salt 'lxb*' saltutil.kill_job $jid

Monitoring

RPM packages are provided by Packagecloud:

https://github.com/lest/prometheus-rpm
https://packagecloud.io/prometheus-rpm

# download the packages from packagecloud
wget --content-disposition https://packagecloud.io/prometheus-rpm/release/packages/el/7/prometheus2-2.2.1-1.el7.centos.x86_64.rpm/download.rpm -P /tmp
wget --content-disposition https://packagecloud.io/prometheus-rpm/release/packages/el/7/node_exporter-0.15.2-1.el7.centos.x86_64.rpm/download.rpm -P /tmp
wget --content-disposition https://packagecloud.io/grafana/stable/packages/el/6/grafana-5.1.3-1.x86_64.rpm/download.rpm -P /tmp
# upload the packages to the local repository
vm sy lxrepo01 -r -D /tmp/{prom,node,graf}*.rpm :/var/www/html/repo/
vm ex lxrepo01 -r createrepo /var/www/html/repo

Prometheus configuration:

Node	SLS	Description
lxmon01	prometheus.sls	Prometheus server configuration
~	prometheus-node-exporter.sls	Nodes expose monitoring metrics with `node-expoerter`

# configure the Prometheus server 
vm ex lxcm01 -r salt 'lxmon*' state.apply
# open the Prometheus metrics page in your default browser
$BROWSER http://$(vm ip lxmon01):9090/metrics
# check the state of nodes defined in for scraping metrics
$BROWSER http://$(vm ip lxmon01):9090/targets

The Prometheus server configuration: prometheus.yml (cf. Prometheus Configuration)

# deploy the node-exporter on all nodes
vm ex lxcm01 -r -- salt -t 120 -C "'* and not L@lxcm01.devops.test'" state.apply prometheus-node-exporter

Grafana configuration:

Node	SLS	Description
lxmon01	grafana.sls	Prometheus server configuration

# open the Grafana web-interface in your default browser
$BROWSER http://$(vm ip lxmon01):3000
# default user/password: admin/admin

Cf. Grafana Configuration:

Prometheus is configured as data-source with prometheus.yml
Import one of the node exporter dashboards, i.e. ID:1860

Usage

Depending on the test the virtual machine resource can be adjusted

# reconfigure the libvirt VM instance configuration, i.e. 2GB RAM, 2 CPUs and VNC support
NODES='lxb00[1-4]' vn co -M 2 -c 2 -vO
# shutdown, undefine, define, start VM instances
NODES='lxb00[1-4]' vn rd

Configuring OpenMPI with a firewall is still an issue:

# switch of the firewall on execution nodes
>>> vm ex lxcm01 -r -- salt -E lxb service.stop firewalld
# Slurm port range Configuration
>>> vm ex lxrm01 -- scontrol show config | grep -i srun
SrunEpilog              = (null)
SrunPortRange           = 35000-45000
SrunProlog              = (null)

Run an example MPI application

# login as a user to an execution node
vm lo lxb001 -r -- su - sulu
# list MPI environments
srun --mpi=list
# compile a sample MPI program
mpicc -O3 /opt/ohpc/pub/examples/mpi/hello.c -o hello
# allocate two nodes
salloc -N 2 /bin/bash
# run the MPI application
srun --mpi pmix hello
# or 
prun hello

Name		Name	Last commit message	Last commit date
Latest commit History 57 Commits
etc		etc
srv/salt		srv/salt
var/aliases		var/aliases
README.md		README.md
source_me.sh		source_me.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etc

etc

srv/salt

srv/salt

var/aliases

var/aliases

README.md

README.md

source_me.sh

source_me.sh

Repository files navigation

OpenHPC Slurm Cluster with SaltStack

Prerequisites

Deployment

Configuration

Package Mirror & Site Repository

SQL Database

NFS Storage

Workload Manager

Monitoring

Usage

About

Releases

Packages

Languages

vpenso/saltstack-slurm-example

Folders and files

Latest commit

History

Repository files navigation

OpenHPC Slurm Cluster with SaltStack

Prerequisites

Deployment

Configuration

Package Mirror & Site Repository

SQL Database

NFS Storage

Workload Manager

Monitoring

Usage

About

Topics

Resources

Stars

Watchers

Forks

Languages