Package Management
There is currently no unified or managed way of extending Csound with additional opcodes, UDOs or CSDs files. The user is required to do this manually by finding extension items online and manually downloading and installing them. This presents a number of difficulties. Users have no centralized way of finding these opcodes and must find the right one for their platform online. Some opcodes might not have a version for every platform leaving gaps in compatibility. There is also no versioning in place to ensure an opcode or UDO works with a given Csound version. There is also a security issue with the current distribution method of files (especially opcodes).
For Csound 7, a new system will alleviate these issues by providing a simple and centralized system for users to find, install or uninstall and update opcodes and other 3rd party items for their platform. It will also foster growth to the number of packages by making it easier for users to contribute.
-
Maintenance: Currently the main Csound repository contains a number of opcodes which contain numerous dependencies and have been a source of issues in relation to coding defects. This places a burden on maintaining the Csound core as building the source is difficult with these added dependencies (particularly for Windows) and also these opcodes have been a source of instability in the system which then requires further attention from core developers. Moving these opcodes out will allow focus on the main core library and reduce the amount of code to maintain.
-
Versioning: There is no versioning of the API to ensure an installed opcode will work with the currently installed Csound version. This can lead to crashes and incompatibilities. The new system should deal with this issue by providing either a manifest file with metadata of the opcode or some other mechanism of interrogating it.
-
Distribution: There is no canonical way of distributing Csound opcodes currently. This means creators are forced to build and host the files themselves while clients must download them from different places and install them onto their machine. This creates an unfriendly user experience.
This section outlines the requirements for a Csound package manager to fulfil. Csound can have a number of extensions to the system. This includes:
- Opcodes (binary data) - allows sharing high performing processing code
- User-Defined Opcodes (UDOs) (text/script file) - allows sharing Csound language-based processing code
- CSD files (text/script file) - allows sharing Orchestra and Score code
Each of these types will need to be managed by one system ideally. The second two have been handled via the CSPM project already (https://github.com/eddyc/cspm). It may be possible to extend this project to allow sharing binary opcodes also.
-
Quarks for SuperCollider is a scripting API and GUI for downloading external modules that are written in the SuperCollider language. The API allows installing by name, GitHub URL or from a local folder. The user is required to rebuild the class library after installation. Each module is stored in an external GIT repository. The Quarks GIT repo contains only a master list text file that contains the module name as well as a git URL that points to the latest head or else a specific tag. When installing a module, Quarks does a GIT clone of the repo to the client system and at the tag given if available. It is possible for the client to save the Quark state like a "package.lock" file in NPM. This will save each Quark and it's version to a file to allow exact reproduction of the modules at a later point.
-
Deken for PureData is a minimal package management system for Pure Data externals. It is a TCL UI application that allows the user to search for packages by name and select an appropriate version for the system. Packages are downloaded via http://puredata.info/ where any registered user can upload an external (what PD calls 3rd party modules). There is no verification by the package manager that the download is from a trusted source. A user is required to manually check the GPG signature to verify. Externals can be PD files or binary data.
- NPM is one of the most well-known package managers and is heavily used by many developers. It allows mostly sharing Javascript code modules and can specify dependencies via a config file (package.json). It allows specifying a global or per-project package repository. It uses https://semver.org/ as it's means of versioning packages. This is a three number versioning scheme (1.0.0) that means essentially "<Breaking API change>.<Backwards compatible feature change>.<Backwards compatible bug fixes>" or "<Major>.<Minor>.<Patch>". This means that only major changes should break compatibility among modules. This has some added benefits such as allowing ranges of dependency versions in the config file.
The following section outlines the end to end process of storing, submitting, building, hosting and deploying 3rd party items to extend the Csound system. It mostly references opcodes currently as they are more complicated. Source files such as UDOs/CSDs will be addressed later.
A number of opcodes will need to be moved out. They are located in the "csound\Opcodes" directory. They are mostly chosen because they have 3rd party dependencies. The list includes:
- CHUA
- Fluidsynth
- WiiMote
- P5GLove
- Faust
- VST4CS
- Linear Algebra
- Python
- Ableton Link
- STK
- CUDA
- OpenCL
- HDF5
- Websockets
- Emugens
- Lua
They will be moved to either the official Csound Opcode GIT repository or else an external GIT repository owned by the opcode author. The official repository will contain a registry file that is the master list of opcodes in the packaging system. All opcodes that are hosted within this repository and externally will be referenced here.
A user can submit a new opcode in two ways:
-
Submit a pull request to the official Csound Opcode repository which contains:
- A new folder with the opcode name.
- Valid C source files for the opcode within the folder.
- Acquire any dependencies depending on the platform.
- A CMakeList.txt file to build the source, dependencies and emit a shared library for each platform.
- Provide an opcode manifest file that details:
- Name of opcode
- Description of opcode
- Version of opcode
- Platforms supported
-
Submit a pull request to update the registry file that points to a GIT repository that contains a valid Csound opcode.
- In this scenario, the opcode must be built by the author and they must provide build artefacts in the GitHub releases or by another undetermined mechanism.
- This method puts more onus on the author to have everything configured correctly with only the registry file acting as a bridge.
TODO
TODO
Any time a pull request is merged into the Csound GIT repository, a build will be kicked off on each CI platform. This includes, currently, Appveyor for Windows and TravisCI for Linux/Mac. This will build the full opcode list. All opcodes should build their dependencies statically so that there is only one output file and no conflict of dependency versions (if any). When the build completes successfully, any new versions of opcodes will be pushed to the binary/package storage service.
Once the opcodes/packages have been built via the CI, they need to be stored somewhere for any client application to download them from. The client application will have to access a registry file from here also to see the catalogue and deal with any versioning and dependency issues.
There are two available solutions:
-
GitHub Releases - The GitHub repo can host release artefacts. In this scenario, the CI would publish any artefacts using the GitHub API after a successful build. This is slightly awkward as there would be many files and for different platforms without any kind of directory structure. It would, however, mean fewer software services to maintain if it was proven to be good enough.
-
BinTray - Is a service that can host files and provide a clean API to access them. This would be the backend for any package manager and can provide a nice directory layout that is easier to reason about. It is another service that needs to be maintained though. Once set up, it may not require much attention going forward, however.
As stated previously, the CSPM application may be adapted to perform these new tasks. Ideally, the application is native to be as responsive as possible. However, that might be too work intensive and a scripting language solution may be adequate. A Node app (like how CSPM is) compiled to a native app via https://github.com/zeit/pkg may be good enough.
The app must provide at a minimum these commands:
-
List - List the current catalogue of opcodes that can be installed. This should be the list of opcodes that are relevant to the current platform or alternatively show all and indicate which are valid for the platform. "Valid" would take into account versioning as well as any other relevant factors.
-
Init - Create a template for a new Opcode/CSD/UDO in the current directory. Provide some dummy config and source files for a user to fill in to create new content.
-
Install - Install the given package by name. Automatically installs the latest relevant version. Should also give the option to install a particular version. Example syntax could be "packagename@1.0.1".
-
Remove - Remove the given package by name. Deletes the package folder from the disk.
Although not involved in package management directly, the installer is related. The installers responsibility is to get the initial set of files onto the system. This includes the core library of Csound with whatever base opcodes there are and any other miscellaneous files and applications. The installer will also manage the optional installation of Csound Frontends. It should also upgrade these files when new releases are available.