Resetting System.CommandLine

[KathleenDollard]

We are resetting the System.CommandLine project to better align with the current ecosystem, isolate parsing behavior so it can be shares with other parser libraries and clarify long term maintenance. This work is intended to bring System.CommandLine out of preview mode.

System.CommandLine provides command line parsing focused on providing a full featured strongly typed command line parser. It is used by the .NET CLI, many .NET tools, and other console applications. The effort has been primarily volunteer powered and changes are needed to shift maintenance responsibility to the .NET teams and align with changes in the ecosystem.

Going forward, the effort will have three parts:

• A core parser which we plan to locate in the .NET Libraries
• An open source full featured parser relying on the core parser and extensible subsystems for end user behavior such as help
• An open ecosystem to encourage community creation/extension of parsers

We will separate the core parser which offers fast POSIX parsing, while also supporting other parsing styles including most Windows style parsers. This parser will be used by a full featured parser that will provide automatic help, tab completion, error reporting and other end-user features. The core parser itself will not provide any end user features, such as help.

A subsystem layer on top of the core parser will support full featured parsing with individually replaceable components. For example, it will support a help subsystem, and we will release the help system we use in the .NET CLI. CLI authors could select a different help subsystem within this extensibility model. We hope this design will foster creativity in the ecosystem by allowing folks to focus on specific aspects of the CLI experience while reusing the rest of the parser.

Parsers today

System.CommandLine is built on preview .NET CLI parsers and has had thousands of PRs across many years to match Posix guidance and user expectations. System.CommandLine has also been updated for fast parsing. While parsers seem simple, there are many hundreds of nuanced expectations within Posix, and parsing Windows command lines remains important for some tools.

For .NET, in addition to parsing behavior end users have expectations on the format of input and how those values are converted to strongly typed values. These two aspects - parsing behavior and type conversions - are the portion of System.CommandLine that is most stable and fast following performance work last year.

In the last five years, the ecosystem has done interesting things with command line libraries. For example, Spectre.Console has beautiful output and a CLI parser. We have worked with Spectre.Console's primary maintainer, @patriksvensson, who would like Spectre.Console's parser to have a dependency on a .NET-provided core parser, shifting responsibility for consistent POSIX parsing to .NET's core libraries. Working the other way, MSBuild is considering taking a dependency on Spectre.Console for output rendering.

A full featured parse will add a layer that meets end-user expectations for help, tab completion, error reporting and other supporting behavior. It will also provide the API used by CLI authors to supply information.

We believe the best way to build a ecosystem with consistent behavior and vibrant features is to provide a reusable core parser, an extensible model for features, and an open implementation of the features used in the .NET CLI and .NET tools.

The road here and onwards

System.CommandLine began as a side project for some Microsoft folks and has had many successes. Because we were creating a full featured parser, we combined feature behavior with core parser behavior. The long journey to redesign has been challenging and sometimes painful as we worked to balance needs of the community, .NET teams, the API review board, and the System.CommandLine team. We've delayed sharing plans until we had the first iteration of code isolating the parser and confidence in this design.

System.CommandLine will remain largely a community project. We are now aligned in intent and have the support of the API review board and the .NET Libraries team on the core parser, which we expect will be maintained by the .NET Libraries team. We are in the process of designing details of the subsystem layer. Overall, this effort involves very little new code since it mostly involves moving code. We have great tests across the system to make things easier.

There will be changes to the API for existing System.CommandLine CLI authors. The System.CommandLine preview will remain available so you can switch to the new library at your convenience. We intend a separate project to replace Dragonfruit with a simplified API that supports full features and subcommands.

The new project is code named "Powderhouse" and will live in a separate branch to avoid disrupting the existing stable main. The name comes from Powderhouse Square in Somerville, MA, in the Boston area.

We have a team, but we'd always love more help. One of the great aspects of that isolating the elements makes it easy to get involved. If you'd like to contribute, you'll find more in the main-powderhouse branch and with the Powderhouse label.

[shaggygi]

@KathleenDollard this is great news! any chance with the reboot there could be a related .net community standup to reiterate this topic, introduce the team, approximate milestones/dates, etc.? again, glad to hear there will be focus (on this complex project) and eventually a "released" version. 👍

[KathleenDollard]

That is a great idea. I will see if we can get on the standup calendar, but we decided to skip next week (March) for the standup because several of us have conference commitments.

[KalleOlaviNiemitalo]

Are prerelease packages from the main-powderhouse branch being published to some NuGet feed? I don't see them in https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json.

[shaggygi]

@KathleenDollard I noticed the above link at "in the main-powerderhouse branch" is pointing back to this issue.

[KathleenDollard]

@KalleOlaviNiemitalo We are very early in the process and do not feel ready for a NuGet feed. We do plan to be on that feed when we are a bit further along.

[KathleenDollard]

@shaggygi Thanks! I thought I fixed that when I fixed the Wikipedia link, but messed up

[WeihanLi]

Great news, is there a roadmap and schedule?

[justinmchase]

There has been a ton of commits in the main branch that are not published in a current nuget package. There are major bugs in the last beta4 release, it would be super nice if you could just drop one last release of whats in main before "resetting" and doing a major revamp.

[iSazonov]

Are you considering integration with shells? For example, PowerShell 7? The easiest way for it is probably to have metadata that it can read. (As a bonus, this metadata could be created for native applications in some manifest.)

[KathleenDollard]

@iSazonov

Are you considering integration with shells? For example, PowerShell 7? The easiest way for it is probably to have metadata that it can read. (As a bonus, this metadata could be created for native applications in some manifest.)

That's not in our current plans. I'd love to see an issue on the problems that it would solve and maybe some background on what would be needed as I am not familiar with that space.

We want to walk that thin line that avoids both feature creep and blocking good ideas that could be implemented in the future or as extensions. The core parser will be fairly locked down when we are done, and the extensions pretty powerful.

[KathleenDollard]

@WeihanLi

Great news, is there a roadmap and schedule?

Not yet. We are still working out the architecture.

[iSazonov]

@iSazonov

Are you considering integration with shells? For example, PowerShell 7? The easiest way for it is probably to have metadata that it can read. (As a bonus, this metadata could be created for native applications in some manifest.)

That's not in our current plans. I'd love to see an issue on the problems that it would solve and maybe some background on what would be needed as I am not familiar with that space.

We want to walk that thin line that avoids both feature creep and blocking good ideas that could be implemented in the future or as extensions. The core parser will be fairly locked down when we are done, and the extensions pretty powerful.

The simplest thing I mean is tab completion/IntelliSense in an interactive script.
In bash, the user is forced to create a special helper script to add this functionality.
PowerShell does this automatically for cmdlets and scripts, but requires non-trivial efforts and the use of a separate module for native utilities.
As you can see, there is a problem that the developer of the utility (or someone else) is forced to make additional efforts to integrate his utility into shells. It is clear that this is not trivial and not everyone will do it.
The ideal solution is to have a standard way for any shell to get meta information about the native utility and create this meta information automatically (for example, when compiling the utility in any programming language).
I guess technically this is close to what you call diagramming, i.e. generating the syntax mapping of the utility. (An example of how this is done in PowerShell is `Get-Command Get-Process - syntax'.) Perhaps it could be a json file that the shell could read. Although there are nuances like how to get this information without running the target utility.

[aetos382]

I agree that the information needed for tab completion should be output to a JSON file at build time.

If the application is run to retrieve information for completion each time the tab key is pressed, then startup performance must be highly optimized.

Run-time performance is important. But I think this tab completion architecture has lowered the priority of features that many people want, but that have a high impact on performance, such as dependency injection.

If we want features like PowerShell's dynamic parameters, the information for completion must be obtained at runtime. But in PowerShell, the cost of doing so would be lighter than launching the application each time, since the host application has already been initialized.

I don't require System.CommandLine to be as flexible as dynamic parameters, but I would like to see mutually exclusive parameter combinations (equivalent to PowerShell's parameter sets) realized. That would be possible based on the information in the static file that was output in advance.

[KalleOlaviNiemitalo]

@aetos382, what do you think would be a good way for shells to locate such a JSON file?

• Extract from within the executable file? That would need parsers for each executable file format (PE, ELF, Python, JAR, etc.).
• Append e.g. .json to the path of the executable file? I don't think distros would want to place non-executable files in /usr/bin/.
• Append e.g. .json to the name of the executable file and look up the directory in a configurable map (e.g. /usr/bin/ to /usr/share/dotnet-suggest/, /opt/smokey/bin/ to /opt/smokey/doc/).
• Append e.g. .json to the name of the executable file and search for that name in directories listed in an environment variable (e.g. DOTNET_SUGGEST_PATH). Judging from macOS dotnet CLI path for global tools is bad - Installer issue sdk#23165, it would not be easy for .NET SDK to set up such an environment variable.

If new features are added to the JSON schema in the future, and applications start using them, then how can completions be kept compatible with older shells (or shell-specific completion scripts) that don't support these?

• Break compatibility and require users to upgrade the shells?
• Install side-by-side JSON files for each version of the schema? Perhaps in separate directories or with different file name extensions.
• Place the new information in new properties within the JSON object, but keep the old properties for older shells?

[KalleOlaviNiemitalo]

A core parser which we plan to locate in the .NET Libraries

Will it support being used in applications that target .NET Framework and are implemented in C# 7? Between #1882 (comment) and #2213, I'm not sure how important this project now considers such compatibility.

[SuperJMN]

I'm very interested in this project. I hope it has a having a brilliant future. I'd like to know if there's any documentation since now there's only a branch, and I guess that's because it's WIP that cannot be trusted.

Anyhow, my best wishes for you.

[KathleenDollard]

Thank you @SuperJMN

You are correct. It is a very rough WIP right now, and we are not ready for people to use it.

[Romfos]

I think that .NET Framework support is a mistake.
Better to have focus on modern .NET with Source Generators, Trimming, AOT, Nullable Reference Types, records, Init properties, e.t.c then "optimized for legacy versions"

if you need .NET Framework you can use alternatives from nuget

[KathleenDollard]

If we fail to support .NET Framework, we are not only cutting out all .NET Framework focused apps, but all the libraries that multi-target. .NET Framework is still a supported platform and we will include it. Since we are using an updated version of C# this should not negatively affect our API.

(It can be confusing that higher C# versions against lower TFMs is an unsupported scenario but works. Microsoft supported scenarios either work or we will fix them. There are many things with this combination that do not work and will never work because they rely on runtime changes and it's a tricky to figure out what works and what does not work.

This works because the C# compiler is in the SDK and can compile to all lower versions of C#. For example, the .NET SDK 8.0.nmm can compile C# 12, and also C# 7.3 and other versions of C#. The emitted IL is mostly the same, and we rely on tests to ensure we have the correct behavior on both TFMs (and we avoid areas of known nuance like tricky overloads). The C# compiler has code that checks the C# version to determine behavior. )

[jscarle-orbis]

I'm glad to see that this project hasn't been abandoned! 🙏

[ian-buse]

Good to hear .NET Framework isn't being dropped. Might not be the latest and greatest, but are still apps that use it for one reason or another.

[shaggygi]

Found this community standup that discusses related... https://www.youtube.com/watch?v=SzpFCVg3JbA