Request for comment: Spack package for GeNN

[muffgaga]

Hey *,

during EBRAINS CodeJam 12 I had some time to work on getting GeNN into the EBRAINS software environment. We use spack as a build-from-source package manager. (This was triggered via @schmitts and @bcumming.)

Here's my first shot at a package file for GeNN:

from spack import *


class Genn(Package):
    """GeNN is a GPU-enhanced Neuronal Network simulation environment based on
    code generation for Nvidia CUDA."""

    homepage = "https://genn-team.github.io/genn/"
    url      = "https://github.com/genn-team/genn/archive/refs/tags/4.6.0.tar.gz"

    version('4.6.0', sha256='5e5ca94fd3a56b5b963a4911ea1b2130df6fa7dcdde3b025bd8cb85d4c2d3236')

    depends_on('gmake')
    depends_on('gcc@4.9.4:')

    def install(self, spec, prefix):
        install_tree(self.stage.source_path, prefix)

    def setup_run_environment(self, env):
        env.append_path('CPLUS_INCLUDE_PATH', self.prefix.include)

This basically just mentions the latest release and its dependency on make (Makefiles are generated by the genn create project thingies) (but ignores the dependency on a recent C++ compiler and the optional dependency on cuda). It installs everything from the release tar-ball and provides CPLUS_INCLUDE_PATH to the user (e.g. via spack load genn or when using spack-generated module files; when another spack package depends on genn we should probably also add a def setup_build_environment(…)).

Now one question regarding GeNN's headers: I saw that you use relative paths in the #include statements for the GeNN headers in the examples. Is there a reason for not setting CPLUS_INCLUDE_PATH? You do mention extending PATH to include GeNN's bin/ folder though…

Cheers,
Eric

[neworderofjamie]

Hey Eric, is the aim to setup GeNN for building models in Python or C++?

[muffgaga]

I started simple and ignored the pygenn stuff 😬… I can try to have a look at the Python library :).

It seems that it's a modified standard Python library install flow: I need to run some make-based thing first, only then setup.py — and I'm also not 100% sure if I can just run install instead of develop.

[neworderofjamie]

Ok, so for C++ all you should need to do is:

• Set CUDA_PATH (potentially OPENCL_PATH) environment variables
• Run top-level make make install PREFIX=XXXX
• Add XXX\bin to the path

Then genn-buildmodel.sh should work correctly and users can build C++ models. As you say the python install is indeed a slightly modified setuptools process. install does work (give or take annoying bugs)

Finally, for now, on Linux, compiler requirement is GCC 4.9.1 (when they fixed std::regex)

[muffgaga]

Thanks for the details — I just tried to follow http://genn-team.github.io/genn/documentation/4/html/d8/d99/Installation.html (the Linux part) and missed, e.g., swig ;))… now I'm iterating spack install genn until it looks okay ;)

[neworderofjamie]

Yeah, the top-level make suggestion was based on my vague understanding of what you might want to be doing in a spack install

[muffgaga]

BTW. spack's cuda package provides CUDA_HOME instead of CUDA_PATH… So we probably need to workaround and have a variant setting in genn → downside: the switch between cuda and non-cuda would have to happen at genn's installation time…

[neworderofjamie]

Ahh that is annoying - CUDA_PATH is NVIDIA's standard I think. Which backend GeNN actually uses is controlled via genn-buildmodel command line arguments so what is present when you build GeNN dictates what is available (the idea of the top-level install is that you only need read access to the install location). You can also just checkout GeNN and stick bin in the path and let GeNN build stuff when it's required (this is typically how I use GeNN)

[muffgaga]

I added:

    variant('python', default=True, description='Enable PyGeNN')
    extends('python',                          when='+python')
    depends_on('swig',                         when='+python')
    depends_on('python@3.8.0:',                when='+python')
    depends_on('py-numpy@1.17.0:',             when='+python')
    depends_on('py-six',                       when='+python')
    depends_on('py-deprecated',                when='+python')
    depends_on('py-psutil',                    when='+python')
    depends_on('py-importlib-metadata@1.0.0:', when='+python')

    # in case of building with PyGeNN we need to build a C++ library
    @when('+python')
    def build(self, spec, prefix):
        make('DYNAMIC=1', 'LIBRARY_DIRECTORY={}/pygenn/genn_wrapper/'.format(self.stage.source_path))
        super(Genn, self).build(spec, prefix)

→ seems to build :).

You can also just checkout GeNN and stick bin in the path and let GeNN build stuff when it's required (this is typically how I use GeNN)

In my initial tests I just installed (copied via install_tree) everything from the source directory to some target folder (spack defines a prefix folder for each package instance (there's a hash computed based on package version, dependencies and variant settings) — that might be a bit excessive. Do you have a suggestion here? Mmaybe skip userproject and don't copy pygenn but only use setup.py install for this?

def install(self, spec, prefix):
    # python setup.py install
    super(Genn, self).install(spec, prefix)
    # the non-python things
    install_tree('bin', prefix.bin)
    install_tree('include', prefix.include)
    # TODO: what else?

[muffgaga]

@neworderofjamie Does the Python library require built-time selection of CUDA/non-CUDA or is it somehow still possible to switch at runtime? (Sorry, I only used GeNN at some hands-on sessions and basically forgot everything :p…)

[neworderofjamie]

So PyGeNN has an inbuilt priority list (CUDA->CPU->OpenCL as OpenCL is still a bit experimental) which, by default, it uses to select from amongst the backends it was build with but this can be override with the backend kwarg to the GeNNModel constructor

[neworderofjamie]

userproject is a bit of a mess as it also includes some helper header files you can use in your own C++ simulation loops so, honestly, copying the whole tree is probably the simplest and best idea

[muffgaga]

So PyGeNN has an inbuilt priority list (CUDA->CPU->OpenCL as OpenCL is still a bit experimental) which, by default, it uses to select from amongst the backends it was build with but this can be override with the backend kwarg to the GeNNModel constructor

Nice! Is this is still possible if we didn't have CUDA at build-time of the dynamic library?

[muffgaga]

userproject is a bit of a mess as it also includes some helper header files you can use in your own C++ simulation loops so, honestly, copying the whole tree is probably the simplest and best idea

I now installed like this:

    def install(self, spec, prefix):
        super(Genn, self).install(spec, prefix) # python setup.py install
        install_tree('bin', prefix.bin)
        install_tree('include', prefix.include)

and after replacing the relative-include I can build some stuff in userproject:

diff --git a/userproject/include/generateRun.h b/userproject/include/generateRun.h
index 84fd3525a..14994563c 100644
--- a/userproject/include/generateRun.h
+++ b/userproject/include/generateRun.h
@@ -17,7 +17,7 @@
 #endif
 
 // CLI11 includes
-#include "../../include/genn/third_party/CLI11.hpp"
+#include "genn/third_party/CLI11.hpp"
 
 //------------------------------------------------------------------------
 // GenerateRunBase

(I believe providing include paths to the compiler via CPLUS_INCLUDE_PATH is reasonable ;).)

However, generator seems to be missing for genn-buildmodel.sh, so I tried

        mkdirp(prefix.src.genn)
        install_tree('src/genn/generator', prefix.src.genn)

However, generator wasn't build in my build ­— did I miss something?
Hmm, I didn't add make('PREFIX={}'.format(prefix), 'install') yet… so maybe this one :)… testing.
Seems I need src/genn completely… testing.

Thx

[muffgaga]

This seems to work:

from spack import *


class Genn(PythonPackage):
    """GeNN is a GPU-enhanced Neuronal Network simulation environment based on
    code generation for Nvidia CUDA."""

    homepage = "https://genn-team.github.io/genn/"
    url      = "https://github.com/genn-team/genn/archive/refs/tags/4.6.0.tar.gz"

    version('4.6.0', sha256='5e5ca94fd3a56b5b963a4911ea1b2130df6fa7dcdde3b025bd8cb85d4c2d3236')

    depends_on('gmake')
    conflicts('%gcc@:4.9.3')

    # TODO: maybe build-time select of cuda?
    variant('python', default=True, description='Enable PyGeNN')
    extends('python',                          when='+python')
    depends_on('swig',                         when='+python')
    depends_on('python@3.8.0:',                when='+python')
    depends_on('py-numpy@1.17.0:',             when='+python')
    depends_on('py-six',                       when='+python')
    depends_on('py-deprecated',                when='+python')
    depends_on('py-psutil',                    when='+python')
    depends_on('py-importlib-metadata@1.0.0:', when='+python')

    # in case of building with PyGeNN we need to build a C++ library
    @when('+python')
    def build(self, spec, prefix):
        make('DYNAMIC=1', 'LIBRARY_DIRECTORY={}/pygenn/genn_wrapper/'.format(self.stage.source_path))
        make('PREFIX={}'.format(prefix), 'install')
        super(Genn, self).build(spec, prefix)

    def install(self, spec, prefix):
        super(Genn, self).install(spec, prefix)
        install_tree('bin', prefix.bin)
        install_tree('include', prefix.include)
        mkdirp(prefix.src.genn)
        install_tree('src/genn', prefix.src.genn)

    def setup_run_environment(self, env):
        env.append_path('CPLUS_INCLUDE_PATH', self.prefix.include)

Do you want to maintain your spack package file in-repo or should I just try to get this into EBRAINS (and upstream spack)?

[neworderofjamie]

I think including it in the GeNN repro makes sense - do you want to make a PR with it named whatever is standard? I guess it would work as is if CUDA_PATH is already set, it just doesn't do anything smart if CUDA_HOME is set by spack?

[muffgaga]

I think including it in the GeNN repro makes sense - do you want to make a PR with it named whatever is standard? I guess it would work as is if CUDA_PATH is already set, it just doesn't do anything smart if CUDA_HOME is set by spack?

To enable a smooth cuda usage we could add:

# …

    variant('cuda', default=True, description='Enable CUDA support')
    depends_on('cuda', when='+cuda')

# …

def setup_run_environment(self, env):
    # …
    env.append_path('CUDA_PATH', self.spec['cuda'].prefix)

[muffgaga]

For arbor @schmitts integrated the spack build into the CI flow of arbor — here it's Jenkins… so probably not as easy for contributors to perform :)?

[neworderofjamie]

Haha, nothing is easy with Jenkins. Also, unlike @schmitts, I don't really understand how spack works so it's not 100% clear to me what the integration would achieve. All it does is copy some files right?

[muffgaga]

[Jenkins rant]

:p (and yes, we also use Jenkins a lot… and there are many problems, but we still don't see a reasonable alternative — at least for situations where you need on-prem. CI with access to local clusters or custom hardware).

I don't really understand how spack works so it's not 100% clear to me what the integration would achieve. All it does is copy some files right?

It also performs the make calls and and setup.py install. A spack build CI job could detect problems on newer cuda or compiler versions — in spack this is merely a modifier for the install call:

$ spack install genn ^gcc@11.2.0 ^cuda@11.4.2
# will install using gcc@11.2.0 and cuda@11.4.2

and you could still test-install in a different variant using an older compiler:

$ spack install genn~cuda ^gcc@10.3.0 
# will install 

Additionaly, spack also supports testing… so we could also run tests on this build. So it's just a structured way of building (and testing) software in multiple variants/versions/settings.

[neworderofjamie]

Unless someone suddenly decides to donate us lots of cloud-time to run workers, we're in the same situation.

That is interesting....Testing against more compiler and CUDA variants is something we want to integrate into our testing system but currently my plan had not got further than using NVIDIA Container Toolkit and docker. Isn't installing gcc from source going to be horribly slow though?

[muffgaga]

Building gcc takes some time, yes — but spack supports build caches… so it's just initial "cost" until the cache has been filled.

Worst case is probably what we do for BrainScaleS: We use spack to provide a containerized software environment comprising ~900 software packages (everything from gcc, to neovim incl. typical simulators and software development tools — ok, excluding texlive because we want to stay below 10GiB for the image file :p) and that typically takes 24h on a 16-core / 64GiB machine to build from scratch.

[muffgaga]

Created PR #486… close this one?