deploy: automatic version bumps

[bbpbuildbot]

New script that will automatically create new versions of certain packages

[heerener]

Used curl to file the pull request to validate the parameters, with the API key that will be used by the job.

[bbpbuildbot]

Thank you for your pull request!

Should you want to clear the PR build directory after failures, please use this pipeline. Before running the cleanup pipeline, please ensure that any PR building pipelines have been cancelled or finished running.

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[olupton]

Just had a quick skim through.

I'm not sure if you considered this, but note that we have some similar logic for manipulating recipes - implemented as a custom Spack command - in https://github.com/BlueBrain/spack/blob/develop/bluebrain/spack-scripting/scripting/cmd/configure_pipeline.py; we use that for configuring Spack in CI pipelines. One still has to fudge the recipe .py somewhat manually, but you would avoid having to parse the recipes to extract the versions and so on.

Up to you.

[heerener]

Just had a quick skim through.

I'm not sure if you considered this, but note that we have some similar logic for manipulating recipes - implemented as a custom Spack command - in https://github.com/BlueBrain/spack/blob/develop/bluebrain/spack-scripting/scripting/cmd/configure_pipeline.py; we use that for configuring Spack in CI pipelines. One still has to fudge the recipe .py somewhat manually, but you would avoid having to parse the recipes to extract the versions and so on.

Up to you.

I hadn't seen that specific code, but it does answer my initial "how do I get the package object in a clean way". Replacing the current code with something similar is definitely possible and gets rid of a regex and makes get_latest_spack_version somewhat simpler, so I'm in favor of it.

I'm still reading the source to insert the new version though

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[bbpbuildbot]

To test your PR, use the following on BlueBrain5:

unset MODULEPATH
. /gpfs/bbp.cscs.ch/ssd/apps/bsd/pulls/2097/config/modules.sh
module load unstable

[matz-e]

I think it would be good to have this be a proper Spack extension in the scripting part. That would also allow to quickly check versions on BB5 manually.

[matz-e]

This could be done with the same logic we already have in gitlab-pipelines, outside of Python. Then we don't need to know about tokens and all that in the Python part (and don't have to modify git URLs)

[matz-e]

could be a regexp... r"^( )+version\(". But actually: we don't want to add version directives into nested statements. Those are normally indicative of a programmatic approach to version bumping. Thus a fixed indent of 4 should be checked and used?

[matz-e]

Wouldn't be needed if this were a proper extension.

[matz-e]

So... Why not do it all in Spack?

Could query our repositories for packages and updates like this:

#!/usr/bin/env spack-python

import re

from spack.version import Version
from spack.util.executable import which

VERSION_RE = re.compile(r'(?:([^0-9].+)-)?v?(\d+\.\d+(?:\.\d+)*)$')


def homogenize(s: str):
    if s.startswith("py-"):
        s = s[3:]
    return s.lower().replace("-", "").replace("_", "")

repo = spack.repo.path.get_repo("bluebrain")
pkgs = repo.all_package_classes()
git = which("git")

for pkg in pkgs:
    print(pkg.name)
    if url := getattr(pkg, "git", None):
        most_recent = max([v for v in pkg.versions if not v.isdevelop()])
        candidates = []
        tags = {}
        for l in git("ls-remote", "--tags", url, output=str).splitlines():
            if l.endswith("{}"):
                continue
            _, tag = l.rsplit("/", 1)
            if m := VERSION_RE.search(tag):
                prefix, version = m.groups()
                if prefix and homogenize(prefix) != homogenize(pkg.name):
                    continue
                candidates.append(Version(version))
                tags[Version(version)] = tag
        if not candidates:
            continue
        most_recent_found =  max(candidates)
        if most_recent_found > most_recent:
            print(f"Current version:   {most_recent}")
            print(f"New version:       {most_recent_found} - {tags[most_recent_foun
d]}")
            print(url)

[matz-e]

Oh, and the source of spack checksum could be very helpful for the packages that don't use tags but have something like pypi.