ConvertFrom-Yaml, ConvertTo-Yaml

[DarwinJS]

Would be great to support Yaml natively.

This was also mentioned by @fabiendibot on #3046

It would also be nice if the CMDLets had the goal of cleanly handling conversion of objects that came from XML as it seems like it would be a frequent use case. Maybe some good tests around this conversion?

[ArieHein]

We had a similar discussion from DSC aspect,
allowing us to change json based configuration files, we wanted to have options for modifying xml based files, YAML based files ,INI based files supporting RegEx swaps from within Text Manipulation cmdlets.

Lack of existing support in PS means we have to work hard to get such ability.
It has been on hold pending community contribution, but if it was baked into PS, it would make it much easier for the DSC part as well.

[lzybkr]

When you say natively, do you mean like XML or JSON?

The current thinking is that YAML should not be baked into PowerShell at all, instead it should be a separate module that you can update without picking up a new version of PowerShell.

If YAML were baked into PowerShell like XML, that would be impossible (think [xml]"b")

If we went the JSON route, you'd have cmdlets to work with YAML - so not really baked into PowerShell, but you'd still have the drawbacks of needing to update PowerShell to get YAML updates.

[joeyaiello]

@lzybkr I know we said we didn't want to bring in a new library, but I think this is something we might need to reassess. Ideally, we should also ship the module on the Gallery, but I think a TON of modern scenarios require YAML now.

Maybe not in 6.0 timeframe, but we should talk about it.

[DarwinJS]

@ArieHein - I have some simple functions that save and retrieve a hash array to the registry. Only handle REG_SZ - but for a simple set of settings it is sufficient - let me know if you want a copy.

I mispoke when I said "native" - I primarily meant "built-in" - it wouldn't bother me if they were shipped-in script modules that could be updated.

[iSazonov]

Our first discussion #2109

[DarwinJS]

@iSazonov - ah yes I see!

I noticed the reference to AWS support of YAML on the thread - I have been converting some templates and have found this to helpful: https://github.com/awslabs/aws-cfn-template-flip

[joeyaiello]

@iSazonov thanks for the pointer, I couldn't find it for some reason. Remember it well, though.

In re-reading that original thread, I think we should definitely implement the cmdlets at some point in the future, and ship them in the Gallery. Based on their quality, and people's perceived usefulness (along with some refactoring work we hope to do after 6.0.0), we can make the in-box vs. Gallery-only call.

[iSazonov]

We have Yaml in https://github.com/PowerShell/platyPS

[MattTunny]

yeah this would be awesome to have, ended up using https://github.com/awslabs/aws-cfn-template-flip to convert

[iSazonov]

@MattTunny Welcome to contribute! :-)

[josepmv]

There's a Windows Server user voice openned to vote for it :-)

https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/11088495-out-of-the-box-support-for-yaml-like-csv-xml-j

[Satak]

This should definitely be part of the native PS 6.1 library. So many things these days are in YAML.

[bergmeister]

There are now psyaml and powershell-yaml modules on the PSGallery but both are not even able to round-trip a YAML file from a VSTS build definition. I don't mind if the module is baked into PowerShell or is a module from the PSGallery.

[BrucePay]

I wonder if the core problem here is the clunky way we deploy modules. Today, you have to find, trust and install a module before you can use it. Compare this with the (apparently) slick way that Javascript does var m = require('mymodule'). Maybe we should have some way to do what DSC does but for native PowerShell. In DSC, when a module is referenced in a configuration, it's automatically downloaded and installed on the target node with no manual effort. Making critical but non-core modules available that way should eliminate the "it should be part of core" arguments. And for nodes that are disconnected from the net, we could have a tool that bundled the dependencies in a script into an archive which is then deployed to the target. This is how the Azure DSC resource extension works - there is a tool that scans a script to figure out the required modules then builds a zip file containing everything that is needed and publishes it to a blob. The Azure resource extension then pulls this blob, installs the modules and runs the script.

[bgshacklett]

For something that is this important, I really don't ever want to depend on a third-party library unless I have some way of vendoring it. It's way too easy for third party developers to potentially break entire ecosystems (see https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/).

Broader issues aside, there is currently no good YAML module for PowerShell, as @bergmeister pointed out. This is a must for a language which is heavily focused towards automation. YAML based configuration files are hugely popular now and it's very hard to avoid them even if you don't have to contend with the opinions of a team to do so. Think of the reasoning behind including XML and JSON as core parts of the language. The case for YAML really isn't so different.

[BrucePay]

@bgshacklett From what I've heard from the Puppet guys, there are just no good YAML parsers :-)

[iSazonov]

Is platyPS parser good enough?

@vors Is there simple way to reuse platyPS YAML parser in PowerShell Core repo?

[josepmv]

I prefer the idea of a separate official module in PowerShell Gallery like @lzybkr says because it would be possible to use it in older powershell versions and it could have its own releases. That would be like the sqlserver module. @BrucePay if it were a page in PowerShell Gallery with Microsoft own modules, it would be easier to find and everybody would know that they can trust them.

But I would understand if it were backed into Powershell as XML and JSON.

The important thing is that it exists ConvertFrom-YAML and ConvertFrom-YAML official functions because YAML is a widely used format for configuration files and it shouldn't be a third-party module, as @bgshacklett point out.

I made a blog entry testing and comparing the two modules I've found to work with YAML files: PSYaml and powershell-yaml.

They have different behaviours because internally they're using different objects:

module	mappings	sequences
PSYaml	OrderedDictionary	Array
powershell-yaml	Hastable	List

I think we need a standard ConvertFrom-YAML and ConvertFrom-YAML.

[gaelcolas]

Actually, ConvertFrom-Yaml in powershell-yaml uses OrderedDictionary when converting with the -ordered parameter.
I've been using this module successfully for a while (in my Datum module for DSC Configuration data, and with kitchen yamls), but don't have a vsts build definition to test with.

Bear in mind that the right way to call it is: get-content -Raw MyFile.yml | ConvertFrom-Yaml -Ordered (people often miss the -Raw).

I wonder why we'd need a Microsoft official module, putting even more overhead on MSFT and reinventing the wheel... Maybe trying to contribute to an existing one first, add tests to avoid regression, open issues to make sure the owner knows the problems is a better approach...
You know what happens when you're trying to create a standard out of the 99 existing implementations...

And yes it would be better outside the language, I agree that the dependency management could be better, bundling everything in PS is not a solution though.
The broad npm issue is also a failure in process. Fork and re-publish fixed it in no time, building apps out of latest version of the internet was the reason it broke so many live apps.

[markekraus]

I agree with @gaelcolas I think this is better with everyone working with the owners of an existing community module to raise and ensure quality.

I'll just add that tests for such a project should include working with a large number of real-world YAML files for things like AppVeyor, Travis CI, VSTS, AWS CloudFormation, etc. For my own experience with YAML deserilization, I have had little success with one solution working universally and have ultimately had to reinvent the wheel several times. In that sense, I agree with @BrucePay "there are just no good YAML parsers".

[iSazonov]

We are talking about this platyPS module because it is already actively used in PowerShell Help environment. I guess no one from MSFT can tell how good this module is because of Code of Conduct. They can either silently reject it or improve it.
And although we've been talking about this a long time ago I don't see how we could use the components of this module here in a simple way.
Maybe @adityapatwardhan and @SteveL-MSFT will open their plans and timeline especially as the new Help RFC is already in the experiment stage.

[SteveL-MSFT]

My personal view is that I would rather see more community modules succeed and become de facto standard than requiring "official" modules from Msft.

[markekraus]

@iSazonov It is one thing to have a solution that works for serializing/deserializing a well defined schema. It is quite another to have a solution that works in general with all schemas that are compliant YAML.

[iSazonov]

I understand the desire of MSFT to reuse community projects to cut costs. But the situation is, in fact, that MSFT may not make use of so many community projects:

• many have bad code, have no trust
• many projects are one person

MSFT has published Powershell specifications more 10 years ago, but nobody ported it yet until MSFT did.
The OpenSSL Project has existed for many years but nobody ported it to Windows while MSFT has not done this.
MSFT revealed many thousands of API interfaces, but how many of them were ported to Unix?
The interesting thing about why the company launched its project .Net Core rather than reuses Mono?
PowerShell is already a year and a half is an open source project, but I see that in this repository only one person from the community makes systematic contribution in the code @markekraus and only one person makes systematic analysis @mklement0.
I don't think that if we divide the project into parts, then we got more contributions.
I don't think the situation will change tomorrow. I wouldn't count on it.

[gaelcolas]

@Vaccano it works well on Linux.

[jdhitsolutions]

My initial thinking is that YAML support should be standard in PowerShell 7.x. YAML is such an integral language to today's enterprises that its omission is glaring considering support for XML and JSON. I have no problem with a separate module that can also be installed from the gallery, but it should be included by default in a new PowerShell 7.x installation. I know there are organizations that won't consider anything open source, and certainly nothing that doesn't originate or is supported by Microsoft. Given the value of PowerShell as a cloud language, native YAML support seems like the right decision.

I am also less concerned about making the module downlevel compatible. If someone needs YAML conversion support for Windows PowerShell, there appear to be a number of options in the PowerShell Gallery.

[redstrike]

As a normal user, I've installed PowerShell Core and use it in my work.
Our projects' ps1 scripts are not compatible with PowerShell Core. I have switched to the legacy PowerShell to run them.
And it's difficult to install a new module in this constrained work environment.
I don't know how to write PowerShell scripts, and now I have to live with the old PowerShell.
I feel sad because the new shiny PowerShell Core that I'm in love, is not usable in even a simple real-world usage.
Maybe, I have to rewrite those *.ps1 files by using a serious programming language.

[joeyaiello]

@PowerShell/wg-powershell-cmdlets has been discussing this issue, and despite the extremely high demand (and my own personal desire to have strong, canonical YAML cmdlets), we're unfortunately going to have to close it (for now) as Won't Fix. The reasoning is two-fold:

• While the PowerShell package is already fairly large, we're doing our best not to add new dependencies to the package so that we don't continue to grow our size unfettered, and because we believe that pulling modules from the PowerShell Gallery (or an internal trusted Gallery, having been vetted appropriately by an enterprise) should be a standard part of a PS7 workflow.
• As a team, we simply don't have the capacity to build a .NET YAML parser from scratch, nor do we have the capacity to vet, support, and service a 3rd party library for which the security implications are high-stakes.

If .NET were to officially support a YAML library, we (as Microsoft) could potentially look into supporting a YAML module on the Gallery. But until that point, the existing community modules will have to suffice.

[potatoqualitee]

Thanks, Joey. I'd like to file a Feature Request in dotnet but don't understand it well enough to know which repo to add the issue. Can you please suggest which repo would be most appropriate within https://github.com/dotnet ?

GitHub

.NET Platform

Home of the open source .NET platform. .NET Platform has 201 repositories available. Follow their code on GitHub.

[potatoqualitee]

(or could a Microsoft person/PowerShell team member place to the request as it may get a higher priority?)

[iSazonov]

See dotnet/runtime#28414

[potatoqualitee]

Thank you @iSazonov. That doesn't look promising :/ Seems that we do need an advocate from the PowerShell team to create the request, eh?

[iSazonov]

I guess Microsoft has internal conclusion that it is low priority and small area.

You can ask PowerShell team online today on Community Call PowerShell/PowerShell-RFC#299

[ghost]

This issue has been marked as won't fix and has not had any activity for 1 day. It has been closed for housekeeping purposes.

[Vaccano]

This makes me sad.

Powershell says ".Net has to do it for us to do it"
.Net says "It is up to Powershell if they want to do it"

I wonder if some of the higher ups at Microsoft realize the runaround that is going on here.

YAML is important. Having it from a signed Trusted Source is critical for most companies. Powershell has built in restrictions making it hard to use things not signed by trusted sources.

I give up. It seems that the enough dust has been kicked up to obscure this issue. I imagine at some point someone high up at Microsoft will say "What? No YAML support in PowerShell?" Fingers will be pointed, blame laid, excuses made (etc). Then it will get done. But no amount of requests by the community will get this through.

[awakecoding]

Why has this been closed as WONTFIX? Seriously, YAML is so common these days, all the cloud-native stuff uses it. One can do so much with just vanilla PowerShell before they need to import a module, and not being to deal with YAML shouldn't be one of them. The built-in JSON cmdlets is one of the reasons why JSON is to popular in PowerShell: it's there, it's standardized, and it works. I am sure I am not alone defaulting to JSON just for that reason, and thinking twice about using YAML if I don't really need it, as it makes my scripts less portable or standalone.

[JustinGrote]

@awakecoding joey's explanation is here: #3607 (comment)

The Json library, even though it was Newtonsoft, was "blessed" by Microsoft as the official JSON library till system.text.json came around.

It's a bummer for sure, I would love native YAML support as well, but I personally think the reasoning is sound. I think energies should be focused on more easily pulling down dependencies quickly for a script/module.

[StephenHodgson]

Seriously surprised this isn't already included. It is sorely needed in the community and would seem like a obvious inclusion.

[schittli]

It's a shame.
We pay many billions in royalties and still have to beg M$ for many years so that not Amazon or Google don't step in again to solve such "problems".
Yes, it's a shame.

This makes me sad.

Powershell says ".Net has to do it for us to do it" .Net says "It is up to Powershell if they want to do it"

I wonder if some of the higher ups at Microsoft realize the runaround that is going on here.

YAML is important. Having it from a signed Trusted Source is critical for most companies. Powershell has built in restrictions making it hard to use things not signed by trusted sources.

I give up. It seems that the enough dust has been kicked up to obscure this issue. I imagine at some point someone high up at Microsoft will say "What? No YAML support in PowerShell?" Fingers will be pointed, blame laid, excuses made (etc). Then it will get done. But no amount of requests by the community will get this through.

[schittli]

After 5 years, it's time to ask M$$ again to do its homework, especially because we need YAML on azure.

Please vote:
#16785

[mubed]

Please re-open this issue. This feature is really necessary.

[JustinGrote]

@mubed Probably won't happen until MS releases a first party YAML library:
#3607 (comment)

Several YAML modules available on the gallery though:
https://www.powershellgallery.com/packages?q=yaml

PowerShell Gallery | Packages matching yaml

[ronascentes]

Native YAML support in PowerShell is extremely necessary nowadays. We have to remember that PowerShell was primary designed for IT professionals and system administrators which most of cases have to deploy vanilla PowerShell scripts over the production environment because they cannot deploy modules (no internet connection, security constraints, not signed binaries etc). Also the module footprint is a concern. A single 2KB script that connects to SQL Server requires a 50 MB Sql Server module to work. If an script requires more than one module, I cannot imagine how much disk space would be required. NPM packages feeling? I really does not understand this concern of PowerShell team of not changing the standard library. Are PowerShell going to the same Javascript path?

[JustinGrote]

@ronascentes it's pretty simple. There is no first-party Microsoft YAML library (there is a JSON one and hence why its there), so having a first-party YAML support brings all kinds of licensing and whatnot issues until that happens.

Fortunately there's lots of great community modules, my favorite being YaYaML.
https://github.com/jborean93/PowerShell-Yayaml

Just run install-module yayaml and you're good to go.

GitHub

GitHub - jborean93/PowerShell-Yayaml: A YAML parser and writer that uses an Assembly Load Context on PowerShell 7+

A YAML parser and writer that uses an Assembly Load Context on PowerShell 7+ - GitHub - jborean93/PowerShell-Yayaml: A YAML parser and writer that uses an Assembly Load Context on PowerShell 7+

Alternatively, ask the .NET team to ship a yaml library with the BCL or ship a microsoft first-party one. YAML isn't native in dotnet as a whole either.

[Vaccano]

I really does not understand this concern of PowerShell team of not changing the standard library. Are PowerShell going to the same Javascript path?

@ronascentes - It is two parts of Microsoft pointing the finger at the other. The PowerShell team says that they need a YAML parser built into .NET before they can add native YAML support to PowerShell. The PowerShell team says it does not have the manpower to build and maintain a .NET YAML parser, and thus cannot add native YAML support PowerShell. (They are unwilling add native YAML support based on a community YAML parser. This is due to licensing and support issues.)

The .NET team says there is not a need for a native .NET YAML parser. They site that there is a fine community YAML parser already out there. Because of this, they are unwilling to build a YAML parser into .NET. They say that if PowerShell needs a YAML parser, they need to build and maintain their own parser. (Which the PowerShell team says they cannot do.)

So basically, it is one company (Microsoft) acting as two companies (PowerShell and .NET). The two parts of Microsoft are unwilling to get together and make a plan to fix this situation. Until a high level Microsoft executive looks and say "What?!?! We don't have YAML support?!?! FIX IT!!", we will get no change to this deadlock. (And no native YAML support in PowerShell.)

[StephenHodgson]

The .NET team says there is not a need for a native .NET YAML parser. They site that there is a fine community YAML parser already out there. Because of this, they are unwilling to build a YAML parser into .NET. They say that if PowerShell needs a YAML parser, they need to build and maintain their own parser. (Which the PowerShell team says they cannot do.)

Same thing was said about json, yet we have System.Text.Json today 🤷

[JustinGrote]

@StephenHodgson agreed, and as soon as the dotnet team recognizes that and ships System.Text.Yaml in Dotnet 11 (or whatever), I'm sure a first party PS module will be soon to follow :)

It will take someone thinking that the existing libraries are not "good enough" in the same way it was determined for NewtonSoft. Honestly surprised as much of a Not-Invented-Here culture that exists over there that it hasn't happened sooner.

Related Discussion FYI: dotnet/runtime#83313

[m3nax]

The .NET team says there is not a need for a native .NET YAML parser. They site that there is a fine community YAML parser already out there. Because of this, they are unwilling to build a YAML parser into .NET. They say that if PowerShell needs a YAML parser, they need to build and maintain their own parser. (Which the PowerShell team says they cannot do.)

Same thing was said about json, yet we have System.Text.Json today 🤷

Without JSON patch support...

[JustinGrote]

You could make a module for that using this library.
https://github.com/weichch/system-text-json-jsondiffpatch

GitHub

GitHub - weichch/system-text-json-jsondiffpatch: High-performance, low-allocating JSON object diff and patch extension for System.Text.Json. Support generating patch document in RFC 6902 JSON Patch format.

High-performance, low-allocating JSON object diff and patch extension for System.Text.Json. Support generating patch document in RFC 6902 JSON Patch format. - GitHub - weichch/system-text-json-json...