🪄✨Sprinkle✨

Sprinkle is used to simplify deployment of Python scripts to DTU's High Performance Computing Cluster. It may also work for other languages, but this has not been tested.

✅ Feature list

• Automatic Miniconda installation
• Automatic setup of environment
• Job submission with arguments passed to job
• Interactive stopping of jobs
• View job output, log, and errors
• View job status including CPU and memory usage
• Change job settings
• Export submission script to file

🚀 Installation

On DTU's HPC cluster, simply run the following commands.

wget -O 'sprinkle-installer' 'https://raw.githubusercontent.com/sarphiv/sprinkle/main/bin/sprinkle' && chmod u+x sprinkle-installer && ./sprinkle-installer update && rm -f sprinkle-installer && source ~/.profile && sprinkle update && sprinkle help

Description of the above commands

# Downloads newest version of sprinkle
$ wget -O 'sprinkle-installer' 'https://raw.githubusercontent.com/sarphiv/sprinkle/main/bin/sprinkle'
# Makes the script executable
$ chmod u+x sprinkle-installer
# Runs the installation script
$ ./sprinkle-installer update
# Delete downloaded sprinkle file
$ rm -f sprinkle-installer
# Update environment variables of current shell
$ source ~/.profile
# Run installed sprinkle for final setup
$ sprinkle update
# Display help view
$ sprinkle help

👉 Usage

All commands below should be ❗ run in your project directory ❗.

1. On your local machine
   1. Transfer project directory to DTU's HPC cluster (see FAQ below)
2. On DTU's HPC cluster
   1. sprinkle start and then enter job parameters (such as main script, CPU cores, memory, and time)
   2. sprinkle help (view other commands)

Sprinkle will attempt to automatically detect and setup your packages. The packages may take some time to install, so please be patient. It will only need to be done once. If this fails, check the FAQ below for how to create an environment file.

📖 Frequently asked questions

How do I create the environment file?

There are multiple ways. If no environment file (or requirements file) is specified, Sprinkle will attempt to automatically create one for you. If your script fails because of the above, you may need use one of the methods below. Once you are done, run sprinkle settings and select your environment file.

• Either manually write the environment.yml file (⭐ Recommended method ⭐)
  1. Create a new file called environment.yml (can be done with touch environment.yml)
  2. Write your environment file (example below, version numbers can be specified).

     name: new_environment_name
     channels:
       - defaults
       - conda-forge
       - pytorch
     dependencies:
       - pip=22.3.1
       - python=3.10.8
       - pytorch
       - pytorch-cuda
       - torchvision
       - torchaudio
       - pip:
         - scikit-learn
         - tqdm==4.61.1
         - opencv-python

• Or export your existing environment

  conda activate <environment name>
  conda env export > environment.yml

• Or create a new environment

  conda create -n new_environment_name
  conda activate new_environment_name
  conda install <package_name1> <package_name2> ...
  pip install <package_name1> <package_name2> ...
  conda env export > environment.yml

Remember the environment.yml file should be created before transfering your project to DTU's HPC cluster. This way you can test your code with your environment on your own computer first. It is much easier to test, find, and fix issues there than on DTU's HPC cluster.

How do I transfer my project to DTU's HPC cluster?

1. On your own computer, navigate through a terminal to the directory CONTAINING your project directory.
   • Example: If your project is in ~/DTU/12345/project_directory, navigate to ~/DTU/012345.
2. Run scp -r project_directory s123456@transfer.gbar.dtu.dk:project_directory
3. Wait for the upload to finish

If the above fails, contact HPC support for more information and guidance.

Where are my script's output, log, and errors?

In a bunch of files in a hidden directory called .sprinkle in your project directory.

My script doesn't work on DTU's HPC cluster.

Check the error, log, and output files with sprinkle view error job_id, where job_id is for your failed job.

The error, log, and output files can also be found in a hidden directory called .sprinkle in your project directory.

How do I connect to DTU's HPC cluster?

Use ssh s123456@login.hpc.dtu.dk or ThinLinc to connect to DTU's HPC cluster. If you use ssh remember to run linuxsh to not overload the login node. Contact HPC support for more information and guidance.

How do I enable developer mode for sprinkle?

Add a file called DEVELOPER-MODE to ~/sprinkle/tmp/. The next call to sprinkle will switch branches and recreate the environment. Remove the file to leave developer mode upon the next call to sprinkle.

🗔 CLI

Sprinkle streamlines management of LSF jobs.

Project repository: https://github.com/sarphiv/sprinkle

In the below, "spr" may be used as a shorthand for "sprinkle".


Usage:
  sprinkle start [--] [<args>...]
    Submit the job script and pass <args> to job script.
    If settings have not been setup, prompt to set them up.
    If environment has not been setup, sets it up.
    If <args> contains dashes, add the two dashes "--" before <args>.

  sprinkle stop [<job_id>... | -a | --all]
    Stop specific jobs or all jobs.
    If nothing specified, prompt to select job to kill.

  sprinkle view [((output | log | error) [<job_id>])] [-a | --all]
    View output, log, or errors of a specific job.

  sprinkle status
    See overview of job details.

  sprinkle settings
    Set up or change existing job settings.
    
  sprinkle setup [-d | --delete]
    Set up job environment (or recreates it in case of changes).
    If settings have not been setup, prompt to set them up.

  sprinkle export [<path>] [<args>...]
    Export submission script to <path> that passes <args> to the job script.
    If <args> contains dashes, add the two dashes "--" before <args>.
    Defaults to working directory.
    
  sprinkle update
    Update sprinkle to latest version.

  sprinkle [help | -h | -? | --help]
    Show this screen.


Options:
  -h -? --help       Show full help text.
  -a --all           For start, kill all jobs; For view, view full file.
  -d --delete        Delete environment without recreating it.

🧑‍⚖️ Disclaimer

This project is a personal project and therefore not affiliated with DTU.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.