bash support

[dsyme]

Powershell support is great, but millions of people know bash and other unix shells and will expect support for it to be "in the box". Currently you can simulate it via this:

#!pwsh
bash -c "ls ~/.nuget/packages/libtorch-cpu/0.3.52118/runtimes/linux-x64/native"

I'd like this to just be

#!bash
ls ~/.nuget

If you can point me in the right direction I can attempt to add an extension that enables this, at least for Linux. We could also add support for bash on WSL, perhaps separately.

[dsyme]

Note that Python notebooks allow a one line execution using unix shell, e.g.

!ls ~/.nuget

Just to note people will complain about the two-line ceremony needed to get to the shell. We can deal with that question as orthogonal to the basics of getting bash scripting in place

[brettfo]

I'm not sure we can easily support #!bash at the top level, per se; PowerShell support always works because it's included in the tool and works cross platform, and it might not be reasonable to include a full copy of bash that works on any platform (then again, given how friendly MS has been to Linux recently, maybe we can do this).

I am interested, however, in adding some kind of 'native shell' experience that might default to bash on Linux/MacOS (or the user's default shell?) and CMD on Windows. I'm not sure what the language specifier should be; perhaps #!shell? The downside to this is that scripts written this way won't work on other operating systems. E.g.,

#!shell
ls !/.nuget

would only work on Linux/MacOS while

#!shell
dir %HOMEPATH%\.nuget

would only work on Windows.

[jonsequitur]

I think the problem with a "default shell" language is that it won't work cross-platform. Shell-specific magic commands like #!bash or #!zsh would be more stable, but will require pulling in an extension at runtime which I expect would use an already-installed version of the shell rather than installing the shell.

We haven't got a working example yet of adding a new subkernel via a NuGet extension but it should work in theory. I'll get back with an example.

[dsyme]

If you could make it that #!xyz searches for a nuget extension that would be great (maybe you do that already?)

I believe a very short one line syntax is simply necessary to be cognitively competitive with Jupyter+python. People do an awful lot with ! in python data science notebooks - downloading files, munging them, moving them around, etc. Perhaps

#!bash -c echo "hello" 

would be enough but I'm sure you'll get people asking for just ! too. Maybe that can happen somehow.

[dsyme]

As part of this you should likely reserve lines beginning with ! - however I'm not sure if that would caus any problems for any languages, I expect not.

Speaking of which, how do you know what the default language for the cells of a notebook is?

[jonsequitur]

If you could make it that #!xyz searches for a nuget extension that would be great (maybe you do that already?)

Are you suggesting reserving these magic commands and acquiring the shell-specific extensions on the fly via NuGet? Definitely doable.

[jonsequitur]

! would be difficult as a reserved prefix for the same reason that we moved away from %, which is that it introduces ambiguities with the embedded languages. # and #! were already effectively comments in C#, F#, PowerShell, and several other languages.

That said, within .NET Interactive, #!-prefixed magic commands are user-extensible, including the custom command line syntax, so #!bash -c echo "hello" would be straightforward.

The default language for a notebook is mapped to the kernel name, which is why in Jupyter we install three different "kernels". The difference between them is just the command line arguments to dotnet-interactive in the kernelspec, in effect:

> dotnet interactive jupyter --default-kernel fsharp /path/to/connectionfile

[TylerLeonhardt]

I can sympathize with wanting a magic command for running a command in the shell that's available on the system.

@dsyme I'm sure your example is a bit of a simplification of something much larger... but ls is ls in PowerShell.

That is to say

#!pwsh
ls ~/.nuget/packages/libtorch-cpu/0.3.52118/runtimes/linux-x64/native

uses the actual ls executable. Any linux executable on the PATH is available in PowerShell.

But yeah a way to invoke one-off commands in whatever the default shell is would be nice.

The only thing we need to be careful is the feature creep because PowerShell support is more than just a shell/one off commands - you get plotting, and get/set variables from other languages, and other rich features of .NET Interactive.

[SteveL-MSFT]

To expand on what @TylerLeonhardt is saying, pwsh is also just a shell. So if you want to run any native commands (like ls), it should just work. You only need bash if you want to write bash scripting.

[dsyme]

Are you suggesting reserving these magic commands and acquiring the shell-specific extensions on the fly via NuGet? Definitely doable.

Yes, that's the suggestion

That said, within .NET Interactive, #!-prefixed magic commands are user-extensible, including the custom command line syntax, so #!bash -c echo "hello" would be straightforward.

That would be perfect

[tmds]

I am for the first time playing with dotnet interactive notebooks, and I was trying to figure out how I can use bash which lead me here.

I guess when you use #!pwsh that pwsh refers to something known to .NET interactive.

Now consider a unix script:

#!/bin/bash

echo "hello world"

Linux kernel looks at this file and sees the #! at the start. Then it knows to take the rest of the line, and use that to invoke the program and pass the script filename as an additional argument.

.NET interactive could do something similar. When the identifier after #! is not known, it can look if finds an executable that matches the filename, and if so, write out the file and execute it.

@jonsequitur @dsyme @TylerLeonhardt @SteveL-MSFT @brettfo wdyt?

[jonsequitur]

A number of the suggestions in this issue will work, but what's the best approach to making them also work on Windows?

Something we've done for SQL was to add a #!sql breadcrumb:

Could something like this work for #!bash and then on Windows, we could direct people to enable WSL and install a distro if needed?

[tmds]

@jonsequitur part of what I'm suggesting is not to limit this approach to bash identifier, but use the same mechanism like Linux kernel, to also allow other shells to be used.

Could something like this work for #!bash and then on Windows, we could direct people to enable WSL and install a distro if needed?

I'm not familiar with how this stuff works on Windows. afaik bash can be made available in other ways than WSL, like as part of the git installer.

Anyway, I think it makes sense that if some well known programs like bash are missing, the user gets an actionable suggestion.

[tmds]

Maybe there is a desire to make environment variables flow from one bash code section to the next?
That would be technically challenging.

[jonsequitur]

For PowerShell we get persistent session support via System.Management.Automation.PowerShell. I wonder if a bash equivalent exists or whether the PowerShell implementation can be used to communicate with other shells.

[TylerLeonhardt]

PowerShell can call the bash executable.

#!pwsh
bash -c foo

works today.

alternatively, .NET Interactive could maintain a Process object of bash and write/read to/from stdio

[tmds]

It is possible to set up an exit hook that communicates back to the host process. Something like:

#!/bin/bash

function at_exit {
  printenv -0 >&3
}

trap at_exit EXIT

[TylerLeonhardt]

I chatted a bit offline with @jonsequitur about this and suggested looking at how Jupyter does this today where they have an API for turning REPLs into Jupyter Kernels:

https://jupyter-client.readthedocs.io/en/stable/wrapperkernels.html

The logic for the process wrapping might be able to be reused but going down this way will give the users the ability to pick any shell they'd like (as long as someone crafted a subkernel that uses this wrapper API).

Could even have a Python REPL Subkernel in .NET Interactive. Or Node.js 😅

I believe they assume that you have the executable on the file system/in the PATH already.

[gbaydin]

I believe a very short one line syntax is simply necessary to be cognitively competitive with Jupyter+python. People do an awful lot with ! in python data science notebooks - downloading files, munging them, moving them around, etc. Perhaps

#!bash -c echo "hello" 

would be enough but I'm sure you'll get people asking for just ! too. Maybe that can happen somehow.

I'm one of the people @dsyme predicted will be asking for ! :) I would like to express my support for this feature request.

[jonsequitur]

A one-liner of some sort should be possible. To summarize where we are:

• ! as a prefix would conflict with existing syntax, e.g. !someValue is a valid statement in C#.
• #!bash could work, though it will fail on some machines.
  • It's possible we can provide shell completions with this syntax as we do within a PowerShell cell.

Thoughts? Other ideas?

[kpko]

Hi, I was looking for shell scripting support for .NET interactive too. In my case I was especially looking for a way to insert "shell script" blocks instead of "one-line" statements via something like #!bash. We use Notebooks for teaching, and this would open doors to providing interactive notebooks for the dotnet cli, docker, linux shell.

IMHO it would be fine if we had multiple implementations for multiple shells. For example, one default #!shell-kernel that uses the default OS shell but also includes a parameter to specify another non-default shell. So, I could specify something like #!shell -s wsl.exe to open a WSL shell.

On the other hand, something like #!bash would work too. This would open a bash compatible shell on Mac/Linux and gracefully falls back to using WSL on windows. We could show a simple information to the user if we are on Windows without having WSL activated to notify the user.

[chaosdinosaur]

@jonsequitur @TylerLeonhardt Hello! A few projects that I have worked with notebooks on have expressed a lot of good support for using .NET Interactive with Polyglot notebooks but one of the major blockers to fully implementing with them is that bash is not supported.

Has there been any continued development or discussion around this? Thanks!

[chaosdinosaur]

If bash could be added and integrated into the .NET Interactive kernel as one of the cells that you could variable share with. I would love to be able to run bash commands like I can in this: Jupyter kernel for bash