NeSI HPC project Scaffolding Template. This project aims to fasciliate the process of deep learning project deployment.
NeSI's documentation is quite comprehensive. but this template seeks the use of python venvs in order to have better dependency management, environment isolation and code reproducibility.
-
Login into any of
MāuiorMahuikabased on your project resource allocation.you can do:
ssh -J ACC_ID@lander.nesi.org.nz ACC_ID@login.mahuika.nesi.org.nzfor mahuika.ssh -J ACC_ID@lander.nesi.org.nz ACC_ID@login.Maui.nesi.org.nzfor Māui. -
git clone https://github.com/alifzl/NeSI-Project-Template && cd NeSI-Project-Template -
deploy your code in
main.py. do not delete theif __name__ == "__main__":statement. -
modify the python version and add desired dependencies in
environment.yml. -
enter your NeSI account ID in
.env. mind the double qoutes. -
modify the first 10 lines of
project_name.slwith your specifications like project ID, resources, etc. -
rename the
project_name.slslurm file to your desired. -
Prepare the Environment by doing:
module --force purge && module load NeSI Miniconda3source $(conda info --base)/etc/profile.d/conda.shconda env create -f environment.yml -p ./venv
this will create a
venvfolder in the project directory. -
sbatch project_name.sl -
check the status by
make my.
after a successfull submission of task, you can use make my command in order to check the current status of your tasks.
if you want to make a change in source code, dependencies, etc. you will be need to do as below:
-
cancel currently unwanted running tasks, if any, by:
make cancel_all -
make cleandeletes any existing venv folder -
module --force purge && module load NeSI Miniconda3 -
source $(conda info --base)/etc/profile.d/conda.sh -
conda env create -f environment.yml -p ./venv -
sbatch project_name.sl
This repo is highly inspired by NeSI's templating example.