[Manual] Devito on Isambard's GPUs and CPUs.

Connect to Isambard login node (https://gw4-isambard.github.io/docs/user-guide/jobs.html)

ssh username@isambard.gw4.ac.uk or just ssh login.isambard

Connect to a login node

ssh login-01

Start an interactive job on a P100 or V100:

qsub -I -q pascalq -l select=1:ncpus=16:ngpus=1
# or
qsub -I -q voltaq -l select=1:ncpus=16:ngpus=1

### Check out the GPU using lscpi [Optional]
lspci | grep 'NVIDIA'
06:00.0 3D controller: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] (rev a1)
81:00.0 3D controller: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] (rev a1)

Then:

Activate conda so that python3.6 is loaded

source /opt/anaconda/bin/activate
python3 --version
Python 3.6.8 :: Anaconda, Inc.

Clone and install Devito via Conda

git clone https://github.com/devitocodes/devito.git
cd devito
conda env create -f environment-dev.yml
source activate devito
source

Load PGI compilers and CUDA toolkit

# module use /lustre/software/x86_64/modulefiles/
# module load nvhpc/20.11
module load pgi/64/19.10
module load cuda10.2/toolkit/10.2.89

Set the following environment variables

export DEVITO_LANGUAGE=openacc
export DEVITO_PLATFORM=nvidiaX
# export DEVITO_ARCH=pgcc
export DEVITO_ARCH=nvc
export DEVITO_LOGGING=DEBUG #optional

You are ready to run a Devito operator on GPU

python3 examples/seismic/acoustic/acoustic_example.py  -d 256 256 256 --tn 128

CPU

Start an interactive job on a Cascade Lake:

qsub -I -q clxq -l select=1:ncpus=40 -l Walltime=03:00:00

Start an interactive job on AMD ROME:

qsub -I -q romeq -l select=1 -l Walltime=03:00:00

Check out the CPU using lscpu [Optional]

[brx-gbisbas@clx-002 pbs.35821.master.gw4.metoffice.gov.uk.x8z]$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    1
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
Stepping:              7
CPU MHz:               800.061
CPU max MHz:           3900.0000
CPU min MHz:           800.0000
BogoMIPS:              4200.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0-19
NUMA node1 CPU(s):     20-39

Then:

Load module files

python3 --version
python3 -m venv clx-env
cd clx-env
source bin/activate
python3 -m pip install -U pip

Clone and install Devito BETTER ON A VIRTUAL ENV!

git clone https://github.com/devitocodes/devito.git
cd devito
python -m pip install -U -e .
module load gcc
gcc --version
// For MPI
# module load openmpi/4.0.4/gcc-9.3
module load openmpi/mlnx/gcc/64/4.0.3rc4
pip3 install mpi4py

Set the following environment variables

export DEVITO_LANGUAGE=openmp
export DEVITO_ARCH=gcc
export DEVITO_LOGGING=DEBUG #optional

You are ready to run a Devito operator

python3 examples/seismic/acoustic/acoustic_example.py  -d 256 256 256 --tn 128

To load htop

module use /lustre/projects/bristol/modules/modulefiles/
module load htop
htop

module load intel/compiler/64/2020/19.1.3

MPI - CPU

• Assuming we already have a python-env with Devito installed

Interactive run

python -m pip install -U -e .
module load gcc
gcc --version
// For MPI
# module load openmpi/4.0.4/gcc-9.3
module load openmpi/mlnx/gcc/64/4.0.3rc4
pip3 install mpi4py

python3 examples/seismic/acoustic/acoustic_example.py -d 256 256 256 --tn 128

Example .pbs script:

#!/bin/bash
#PBS -q clxq
#PBS -lselect=2:ncpus=40:ompthreads=40
#PBS -l walltime=05:00:00

module load gcc
module load openmpi/mlnx/gcc/64/4.0.3rc4

export DEVITO_LANGUAGE=openmp
export DEVITO_ARCH=gcc
export DEVITO_LOGGING=DEBUG #optional
export OMP_PLACES=cores
export OMP_PROC_BIND=close
export DEVITO_AUTOTUNING=aggressive
# export KMP_AFFINITY=granularity=fine,balanced
export DEVITO_JIT_BACKDOOR=0
export DEVITO_MPI=1


source  ../environments/python-venv/bin/activate

ls

mpirun -n 2 --bind-to socket --map-by socket python ../devito/examples/seismic/acoustic/acoustic_e$