Replace settings.json with settings.yaml and provide a new syntax

[Geod24]

This adds parsing for settings.yaml (settings.yml is not supported), which takes precedence over settings.json.
As we are introducing a new feature, this also slightly change the format to be more intuitive, consistent, and less repetitive. The main change is on the way one can specify registries: instead of having two fields, only have one, which might lead to a bit more repetition in some cases, but generally should be more intuitive to users.

Fixes #1832

Still a draft because I want to do a self-review (I noticed a few typos) and I might do changed before this refactoring, notably around environments.

[s-ludwig]

Has this been discussed in general? Since file formats have been a sensitive topic (as with most bike shedding topics), I'd suggest that we reach some kind of conclusion before adding any kind of new format support, to avoid trouble later on. Personally, I really dislike YAML for its awkward complexities, and I also really see no point in introducing yet another format just for a trivial configuration file (or worse, using this as an excuse to introduce a third recipe format).

Having said that, JSON is certainly not ideal for writing configuration files, so I certainly see the point, they would look prettier with YAML. But then I'm thinking that the most complex configuration file that I've written was three lines long. Even XML would be fine for that in terms of overall syntax overhead.

[CyberShadow]

What about SDLang (.sdl)? It seems to be the obvious choice and I don't see it mentioned here or in the DFL meeting notes.

[veelo]

@CyberShadow from #1823:

JSON has proven to be a bad format for human-written configuration. Be it the lack of comments, noisy syntax, lack of typing... So much that a push was done to move to SDLang a few years ago.
However JSON enjoys wide support among editors and tools, and is still familiar to people than SDLang, the later being yet another skill a newcomer to D has to pick up.

[Geod24]

Has this been discussed in general? Since file formats have been a sensitive topic (as with most bike shedding topics), I'd suggest that we reach some kind of conclusion before adding any kind of new format support, to avoid trouble later on. Personally, I really dislike YAML for its awkward complexities, and I also really see no point in introducing yet another format just for a trivial configuration file (or worse, using this as an excuse to introduce a third recipe format).

I've been bringing it up at DLF meetings a few times now. Note that the main issue that led to the reversion of SDLas default was that long thread on the forum: https://forum.dlang.org/thread/gnfctbuhiemidetngrzi@forum.dlang.org
The main point that came up was how obscure SDL is compared to JSON, and that trumps the benefit of SDL being a "better" format. YAML is both very well known, and much better than JSON.

I heard the "awkward complexities" argument a lot. But I don't really buy it. The subset of features we use for every Dub configuration files is very simple.

Having said that, JSON is certainly not ideal for writing configuration files, so I certainly see the point, they would look prettier with YAML. But then I'm thinking that the most complex configuration file that I've written was three lines long. Even XML would be fine for that in terms of overall syntax overhead.

It's chicken and egg problem. I don't want to extend uses of this configuration file before it can look decent.

[s-ludwig]

I heard the "awkward complexities" argument a lot. But I don't really buy it. The subset of features we use for every Dub configuration files is very simple.

@Geod24 Honestly, the same argument works at least as strong for SDLang. And the complexity of the different available DUB directives is much larger than the file format itself, especially for the subset that is used. On the other hand, even the reduced subset of YAML that is typically used includes many redundant syntaxes that you have to understand (JSON, quoted or quote-less strings, indentation based lists and maps, indentation style differences) where the only argument is that someone might already know them because they are using one of the popular CI or container services or another language that uses YAML. However, I would bet that, contrary to the core community, most general D users have had only vague to none experience with YAML - unfortunately that is hard to measure.

But where I really can't follow you is when you say that you have to constantly look something up for SDLang because of its syntax, but at the same time immediately remember the same thing for JSON. I could definitely say the same for JSON for the more obscure places (e.g. custom build types), but that's not a language syntax issue, it's simply not knowing how the DUB directives are mapped to JSON (or SDLang for that matter).

[Geod24]

However, I would bet that, contrary to the core community, most general D users have had only vague to none experience with YAML - unfortunately that is hard to measure.

I think that's where our opinion differs. YAML is everywhere. More importantly, it is in the technologies that new generations of programmers are learning (e.g. K8s). Not to mention, any popular editor is likely to come with built-in JSON support and probably built-in YAML support. Not that I care, I use emacs from my terminal, but I am certain that there is a large portion of users out there that care.

But where I really can't follow you is when you say that you have to constantly look something up for SDLang because of its syntax, but at the same time immediately remember the same thing for JSON. I could definitely say the same for JSON for the more obscure places (e.g. custom build types), but that's not a language syntax issue, it's simply not knowing how the DUB directives are mapped to JSON (or SDLang for that matter).

Well there's a bit of muscle memory going on, that much is true. But I still have to lookup how to set a custom dflags on dependency in SDL. Maybe it's just just my problem. But I can already picture how it'd be in YAML, because the format maps more naturally to a structure for me (not because JSON/YAML are better, purely because I use them almost every day).

[s-ludwig]

Just to get this into perspective, this is all of the syntax that we are talking about (maybe add \ line continuations if you want):

// Nodes can have attributes
author "Peter Parker" email="peter@example.org" active=true

// Nodes can be arbitrarily nested
contents {
	paragraph "This is the first paragraph"
	paragraph "This is the second paragraph"
}

[s-ludwig]

So it would be nice if we could agree that we are mostly talking about "simpler, but less known" vs. "more complex, but more known" (how much in is unclear). At the same time complexity of the recipe directives >> complexity of the used syntax. So the whole argument is objectively quite weak. The much larger part here is obviously personal taste/habit and opinion.

Edit: Maybe instead of "objectively", I should have said something like "technically", since opinion and taste can of course still be objectively important from a marketing perspective.

[WebFreak001]

But where I really can't follow you is when you say that you have to constantly look something up for SDLang because of its syntax, but at the same time immediately remember the same thing for JSON. I could definitely say the same for JSON for the more obscure places (e.g. custom build types), but that's not a language syntax issue, it's simply not knowing how the DUB directives are mapped to JSON (or SDLang for that matter).

I think IDE support greatly helps here. In code-d I had JSON auto-completion for the entire dub.json schema for ages. I have also supported SDL for nearly just as long time with a custom parser & validator I wrote in typescript to basically use the same JSON schema with just very few modifications. How simple SDL is as a format made the 100% implementation quite easy. I don't know how much I'll be able to support YAML in that regard and how long implementing it would take if we decided to make a dub.yml recipe variation as well.

Also, even if still in development, I think https://docs.webfreak.org/dub-reference/build_settings/ is a great reference for users and we should extend that a bunch more too.

I'm not really strongly in favor of either side. What's nice about SDL is the simple syntax - easy to implement and read, but still flexible enough to write. Maps well to D's syntax as well.

Personally I don't think it's a disadvantage that it's not really widely used, because it's intuitive enough, but I know some others I personally was showing D to that were annoyed just thinking about having to learn another format, even if it's not really much to learn when you then start to do it. If anything, we definitely need to better market the format. E.g. for starters maybe don't call it dub.sdl, but call it dub.conf or project.dub or something like that. (it doesn't imply anything about the format, sdl sounds too exotic I think)

If rust/cargo was able to push TOML through, which is just extended INI with an imo very similarly sized scope and learning curve to SDL, I think we could do it with DUB+SDL as well.

YAML is a very nice format to write for the user, but it takes an immense amount of code and libraries to get tooling to work nicely with it, if we were to make a dub.yml recipe format for example. (and then you also probably only have a 95% solution, because users with big projects and complex recipes will probably start using some of the useful YAML things such as back-references, which are unlikely to be easily supported outside just parsing) For a config file format I think it's pretty good though, even if we only really use 10% of YAMLs feature for it.

The first time I used YAML I did so with Minecraft plugins in bukkit/spigot, and I had some positive experiences with it there, in a time when XML was usually the default choice in other languages or frameworks. I don't know where YAML is used nowadays or where new programmers would start to pick up on it now, but I'm not sure if it's a great experience if we can't properly show newcomers how to write and use YAML. If we decide to push YAML in DUB more, we should definitely first work on better marketing, tutorials and docs for it. E.g. don't try to push it through before we have docs for everything in place + sample projects + some extras for people to really get hooked on.

What's great about adoption here is that tools that beginners are likely to use, such as GitHub actions, also use the format, which makes it feel like a more worthwhile investment to dive into YAML for anyone starting to program.

[Geod24]

Just to get this into perspective, this is all of the syntax that we are talking about (maybe add \ line continuations if you want):
[...]

One thing that tripped me up a few times is that spaces are not significant but newlines are. E.g. you can't have:

contents { paragraph "Look ma, no newline!" }

Correct ? Or am I misremembering ?

So it would be nice if we could agree that we are mostly talking about "simpler, but less known" vs. "more complex, but more known" (how much in is unclear). At the same time complexity of the recipe directives >> complexity of the used syntax.

Agreed, and that's the point I was trying to make when I mentioned that SDL was less well known than YAML. If you look at the forum thread, you will even see people thinking we invented SDL!

So the whole argument is objectively quite weak. The much larger part here is obviously personal taste/habit and opinion.

Yes and no. The path of least resistance which I considered taking for a while was to go with JSON5. Comments, trailing whitespace, that would make me happy with minimal work. But I believe YAML is the better path for the project (I also considered other formats in the prospect).