Dub Plugins Allowing Common Patterns to be Shared

[vnayar]

When working with dub, often common solutions are discovered for common problems, e.g. setting up integration tests, detecting and building protocol buffer files, etc. As of right now, the only way to share knowledge and usage of such solutions is to copy snippets of dub files into your own project.

As dub configuration solutions get more complex or receive updates, it becomes harder and harder to keep all projects up to date with the latest state of knowledge.

One possible solution would be to introduce a "dub plugin", which allows common configuration to be built and maintained, in a manner similar to maven plugins from Java. These dub plugins would have their own repository that could be downloaded from during build time, and they would have the ability to latch onto the build process.

For example, the Maven Failsafe Plugin takes configurations such as the file naming pattern of integration tests in a project, and during a built, when the verify phase of the build comes, the plugin adds logic to find those test files and run them.

This is possible because Maven exposes a build lifecycle that plugins can attach to and execute code for.

[WebFreak001]

Dependencies can already inject into the build lifecycle (preBuild, postBuild, preGenerate, postGenerate) of your package by being a sourceLibrary, however you can't really configure them too much.

@Geod24 should we allow custom configuration sections in dub recipes for dependencies to read? (e.g. "x-protoBufFiles": [...]) that would avoid cluttering random configuration files for every dependency into the project, but it might also make sense to only have random configuration files as there aren't that many packages that would really make use of this.

We could also move out things like the integrated ddox generator into packages.

[rikkimax]

Unfortunately shared libraries are out, the implementation is still not good enough for that unless you restrict yourself, use specific compilers with specific flags ext.

[Geod24]

Then we fix the implementation :)

[rikkimax]

Please do!

I failed at my attempts for dmd. But at least its well documented how I failed :')

[WebFreak001]

I think having dub fetch is good for how we can get the plugins.

I would probably go with just running executables and giving them some kind of API to read config from dub.

In another project I made I just read from $DUB_PACKAGE_ROOT, using dub convert --format=json --stdout (though I actually didn't do that for JSON files as that invocation was super slow)

I think this is a pretty good way how to do a bunch of stuff, we should just extend it a little bit to be able to do more stuff.

[rikkimax]

For a shared library system, dub will need a configuration for it to be able build shared libraries that are compatible with it. Flags, version of itself to force, and the compiler to use.

From there it is only a matter of telling dub to load a plugin via dubs config or a package file.

[WebFreak001]

another common pattern that we have been discussing for a while is custom sub-commands, e.g. through the registry via some future namespacing or just allowing all packages that start with dub- to be installable, similar to how custom git or dotnet commands can be made.

I think having extensible dub.json/dub.sdl sections like the one with x- would allow such sub-commands to give users configuration options. Additionally such extension options can be considered to be introduced into the core build settings / package recipe documentation. (open issue: how is SDL/JSON conversion handled for such unknown keys?)

From #252:

This is just a thought and might not be a good idea.

Dub has quite some special knowledge, like vibe.d, mono-d, visual-d and rdmd.
This makes it a very handy tool, but it also requires every tool to be added in dub directly.

My thought is to forward these to special binaries; dub-build-* and dub-init-*. This way you can easily extend dub for custom libraries/projects without touching dub itself.
dub-init-monod would thus create a mono-d project etc.

[WebFreak001]

from #304:

I've started writing toD compilers (e.g. .asn1 to .d) to facilitate the development process of certain projects. This involves sending arguments to a compiler executable depending on the build environment. It can also be useful for compiling *.dt to *.d on a per-file basis rather than increasing the compile-time of an entire project/library (also making it possible to have 100s of diet template pages).

I've thought of making a fork of dub to add the json arguments and additional compilation steps but this brought to mind the possibility of having these as plugin features in dub.

• A new json parameter (call it plugins) can select external/internal source dependency(ies) that, when compiled, create a new dub executable that embeds the plugins through template functions
• The newly compiled plugin-enhanced dub is forked and handles the entire compilation process without handling the plugins json parameter

Ideally, the way the plugins are compiled into dub would serve the purpose of facilitating an otherwise complex build process. They would have access to several template function calls on several occasions:

• Before opening the dub.json file a plugin can add a set of additional json identifiers and structures to be recognized
• After parsing the dub.json file a plugin can access the json data
• Before every project is compiled a plugin can be notified
• After every project is compiled a plugin can be notified

This is a minimal feature set that would truly empower dub and allow "sub-compilers" to be added to D projects seamlessly.

---

Here is a more complete description of the whole process, and an outline of a possible implementation. I will be using master, slave and forked to describe the 3 different dub instances involved. The master dub is the one invoked during normal compilation of a project, and the slave dub is the dub source code modified at runtime by the master dub when a plugins field is met in a main project's dub.json.

To be more specific, when the master dub reads the plugins array in the main (project's) dub.json, there's a specific process involved when building the slave dub in order to embed those plugins to be compiled into (a slave) dub itself by an unchanged forked dub, to then be awarded the compilation of the main project.

1. The slave dub.json has the plugins appended to it as dependencies
2. The slave dub source has a new plugins.d file generated by the master dub at runtime by looping through each plugin namespace identifiers and adding a group of template static functions that call the underlying plugins' template callbacks sequentially
3. An appropriate versions item is added to the slave dub.json allowing the rest of the slave's source to import dub.plugins and to call the proper plugin functions at the proper steps of the compilation when it will execute the main dub.json (main project)
4. The slave dub.json is then run through a forked dub to be compiled and then is awarded handling of the main dub.json (main project) now ignoring the plugins field but effectively running them during the compilation process
5. Error handling is done by redirecting stdout appropriately
6. The master dub ends when the forked dub and the slave dub are done

[WebFreak001]

From #743:

Hello! I'd like to run some platform-specific scripts on clean (such as make clean) when running dub clean but can't find a way to hook this into dub.sdl.

For the moment I have make call dub for the clean target, but it's complicated for nothing :)

[s-ludwig]

See also https://github.com/dlang/dub/wiki/DEP3