Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/slurm remote support #250

Draft
wants to merge 30 commits into
base: develop
Choose a base branch
from
Draft

Feature/slurm remote support #250

wants to merge 30 commits into from

Commits on Nov 5, 2020

  1. Move slurm job template to separate SLURM directory

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    60b78d0 View commit details
    Browse the repository at this point in the history

  2. Add parser for new JSON file type, that describes a remote SLURM node

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    68f49fb View commit details
    Browse the repository at this point in the history

  3. Change CLI such that different SLURM node templates can be selected

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    f5ea82d View commit details
    Browse the repository at this point in the history

  4. Add additional fields to Slurm.Job case class 

    Job scheduling logic shall be moved from {Compose,HLS}-Task into the
Slurm object. For this, some additional information is required.
    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    0ba49cb View commit details
    Browse the repository at this point in the history

  5. Add option for remote execution to Slurm object

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    4b01364 View commit details
    Browse the repository at this point in the history

  6. Add a wrapper that handles local/remote execution of shell commands

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    b35dc81 View commit details
    Browse the repository at this point in the history

  7. Add Pre/Postamble that runs before/after SLURM Job 

    Preamble copies all files that are required for the current job to the
SLURM node, postamble copies all generated artefacts back from node.
    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    9683925 View commit details
    Browse the repository at this point in the history

  8. Make writeJobScript() work with remote paths

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    3b390c8 View commit details
    Browse the repository at this point in the history

  9. Adapt Slurm.apply() to remote execution

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    b029b77 View commit details
    Browse the repository at this point in the history

  10. Add postamble that pulls generated files from node

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    5e43cfa View commit details
    Browse the repository at this point in the history

  11. Use explicit identity func instead of custom lambda

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    c54e988 View commit details
    Browse the repository at this point in the history

  12. Postamble: add timing and utilization report

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    92415ba View commit details
    Browse the repository at this point in the history

  13. Fix local SLURM execution

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    3e2183f View commit details
    Browse the repository at this point in the history

  14. Add optional sbatch CLI options to slurm json

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    619893e View commit details
    Browse the repository at this point in the history

  15. Add support for user-defined pre/postamble 

    The absolute path to both scripts may be supplied via the key
"PreambleScript" resp. "PostambleScript" in the SLURM JSON cfg file.
    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    16b4464 View commit details
    Browse the repository at this point in the history

  16. Remove unnecessary func parameter

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    773c47b View commit details
    Browse the repository at this point in the history

  17. Fix: Copy SLURM job script to correct host

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    0d4c832 View commit details
    Browse the repository at this point in the history

  18. Finalize SLURM job template for ESA cluster

    @mhrtmnn
    mhrtmnn committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    c5911ef View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2020

  1. Use Paths.get() instead of Path.of()

    @mhrtmnn
    mhrtmnn committed Nov 16, 2020
    Configuration menu
    Copy the full SHA
    6b01e25 View commit details
    Browse the repository at this point in the history

  2. Use unique slurm script names

    @mhrtmnn
    mhrtmnn committed Nov 16, 2020
    Configuration menu
    Copy the full SHA
    74436f4 View commit details
    Browse the repository at this point in the history

  3. Fix hls postamble

    @mhrtmnn
    mhrtmnn committed Nov 16, 2020
    Configuration menu
    Copy the full SHA
    48c5fde View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2020

  1. Only pull SLURM artefacts if job was successful

    @mhrtmnn
    mhrtmnn committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    6c92d0c View commit details
    Browse the repository at this point in the history

  2. HLS task: check slurm return code

    @mhrtmnn
    mhrtmnn committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    27d8af3 View commit details
    Browse the repository at this point in the history

  3. Handle return value correctly in slurm job

    @mhrtmnn
    mhrtmnn committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    9d496b3 View commit details
    Browse the repository at this point in the history

  4. Fix vivado hang

    @mhrtmnn
    mhrtmnn committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    3207acd View commit details
    Browse the repository at this point in the history

  5. Reduce verbosity, cleanup

    @mhrtmnn
    mhrtmnn committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    1d81cd6 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2020

  1. Cancel SLURM job if tapasco is terminated

    @mhrtmnn
    mhrtmnn committed Nov 28, 2020
    Configuration menu
    Copy the full SHA
    ff4a8e9 View commit details
    Browse the repository at this point in the history

  2. Show duration for which SLURM job has been running

    @mhrtmnn
    mhrtmnn committed Nov 28, 2020
    Configuration menu
    Copy the full SHA
    b7c0a20 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2020

  1. Refactor Slurm tasks into slurm jobs 

    Previously, a job would be broken into its tasks, and a new tapasco
job would be created for each task. These jobs were then executed on the
SLURM cluster. Refactor this, such that the original job is executed on
the SLURM cluster as-is, which simplifies the SLURM logic.
    @mhrtmnn
    mhrtmnn committed Dec 29, 2020
    Configuration menu
    Copy the full SHA
    259e6ee View commit details
    Browse the repository at this point in the history

  2. Adapt Slurm pre/post-amble to changes in 259e6ee 

    Since SLURM cluster now processes whole jobs (instead of single tasks),
dependencies (preamble) and produced artefacts (postamble) of multiple
platform/architecture pairs may need to be transferred.
    @mhrtmnn
    mhrtmnn committed Dec 29, 2020
    Configuration menu
    Copy the full SHA
    f84e5cc View commit details
    Browse the repository at this point in the history