GRASS on Windows with Jupyter Notebooks unable to use ##bash #3195

vishalm1975 · 2023-10-02T19:39:22Z

vishalm1975
Oct 2, 2023

I'm trying to use Grass GIS 8.3.1 with Jupyter Notebooks on Windows 10 Home. and bash. bash kernel is installed. #lsmagic lists bash as a cell magic that is available. I get the following error:

##bash
g.list type=rast

CalledProcessError Traceback (most recent call last)
Cell In[20], line 1
----> 1 get_ipython().run_cell_magic('bash', '', 'g.list rast\n')

File C:\GRASS GIS 8.3\Python39\lib\site-packages\IPython\core\interactiveshell.py:2493, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2491 with self.builtin_trap:
2492 args = (magic_arg_s, cell)
-> 2493 result = fn(*args, **kwargs)
2495 # The code below prevents the output from being displayed
2496 # when using magics with decorator @output_can_be_silenced
2497 # when the last Python token in the expression is a ';'.
2498 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File C:\GRASS GIS 8.3\Python39\lib\site-packages\IPython\core\magics\script.py:154, in ScriptMagics._make_script_magic..named_script_magic(line, cell)
152 else:
153 line = script
--> 154 return self.shebang(line, cell)

File C:\GRASS GIS 8.3\Python39\lib\site-packages\IPython\core\magics\script.py:314, in ScriptMagics.shebang(self, line, cell)
309 if args.raise_error and p.returncode != 0:
310 # If we get here and p.returncode is still None, we must have
311 # killed it but not yet seen its return code. We don't wait for it,
312 # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
313 rc = p.returncode or -9
--> 314 raise CalledProcessError(rc, cell)

CalledProcessError: Command 'b'g.list rast\n'' returned non-zero exit status 127.

Is it possible to fix?
Thanks,
Vishal

wenzeslaus · 2023-10-03T03:38:12Z

wenzeslaus
Oct 3, 2023
Maintainer

GRASS GIS on Windows with Bash, let alone Bash kernel for Jupyter, is not something we ship, so I can't tell what setup you have there exactly. Bash kernel for Jupyter is interesting and I played with that and GRASS GIS in the past (on Linux), but I abandoned that idea in favor of exclamation mark lines (!command line syntax) and %%bash cell magic in IPython kernel which are available out of the box and you can combine them with Python. You need to set up GRASS GIS session if you want to run GRASS GIS commands like that (i.e., without grass ... --exec) and that seemed like suboptimal for me with Bash kernel. There should be no problem doing grass ... --exec g.list rast in Bash kernel. I would first make sure that your Bash kernel actually works on Windows (without running GRASS GIS commands).

2 replies

vishalm1975 Oct 3, 2023
Author

All I'm trying to do is repeat this workflow, which I assume did work on a Windows machine:
https://github.com/ncsu-geoforall-lab/GIS714-assignments/blob/main/GRASS_GIS_Foundations/JupyterOnWindows_Tutorial.ipynb

I installed bash and jupyter lab in addition to what's in the above setup. I think this is a bug, for what it's worth, because it seems to have worked for others according to several tutorials out there , for example, from Helena and Vero's workshops and lessons.

I don't particularly understand much of the technical platform terms. At the same time, I have used GRASS for 23 years, and would like to continue to use it, without having to figure out so many things each time. I am sure there are many like me out there. I used to use GRASS scripts extensively in bash on Linux and did major portions of my MS and PhD this way. Now i'm evaluating whether I can use some of the ways using Notebooks and still use the old familiar ways, but on Windows since no one else uses Linux at my workplace. (In any case I think I'll have a different set of issues if I try Linux.)

The command line syntax does work but grouping many commands together under ##bash does not.
GrassGIS_1.pdf

I'm attaching the notebook and its output as a pdf and hope that can help figure out whether this is a bug or not,

Thank you,
Vishal

wenzeslaus Oct 4, 2023
Maintainer

The PDF is helpful. It shows that bash itself works. Can you run the env command in your %%bash cell (clean up the output before copy-pasting it here in case there is something private) and try some of the exclamation lines.

It looks like the bash does not inherit the environment variables from the kernel. You can test that by setting variables yourself with os.environ in a standard cell and using echo or env in the %%bash cell. You can also compare the behavior between %%bash and %%python.

vishalm1975 · 2023-10-04T19:41:10Z

vishalm1975
Oct 4, 2023
Author

Hi Vaclav, 1. The exclamation lines work, e.g. !g.list rast 2. ##bash env gives me this: HOSTTYPE=x86_64 LANG=C.UTF-8 WSL_INTEROP=/run/WSL/8_interop WSL_DISTRO_NAME=Ubuntu USER=vishal PWD=/mnt/c/Users/15302/Pythonwork HOME=/home/vishal NAME=DESKTOP-OMLIGRO XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop SHELL=/bin/bash TERM=xterm-256color SHLVL=1 LOGNAME=vishal PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/GRASS GIS 8.3/bin:/mnt/c/GRASS GIS 8.3/scripts:/mnt/c/GRASS GIS 8.3/extrabin:/mnt/c/Users/15302/AppData/Roaming/GRASS8/addons/bin:/mnt/c/GRASS GIS 8.3/lib:/mnt/c/GRASS GIS 8.3/bin:/mnt/c/Users/15302/AppData/Roaming/GRASS8/addons/bin:/mnt/c/Rtools/bin:/mnt/c/Rtools/mingw_32/bin:/mnt/c/Rtools/mingw_64/bin:/mnt/c/GRASS GIS 8.3/extrabin:/mnt/c/GRASS GIS 8.3/bin:/mnt/c/GRASS GIS 8.3/Python39:/mnt/c/Program Files/AdoptOpenJDK/jre-8.0.242.08-hotspot/bin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/Pandoc/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Graphviz/bin:/mnt/c/Users/15302/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/15302/AppData/Roaming/TinyTeX/bin/win32:/mnt/c/Program Files/Graphviz/bin:/mnt/c/GRASS GIS 8.3/:/mnt/c/Users/15302/AppData/Local/GitHubDesktop/bin:/mnt/c/Program Files/RStudio/bin:/mnt/c/Program Files/R/R-4.0.3/bin/x64:/mnt/c/GRASS GIS 8.3/:/mnt/c/GRASS GIS 8.3/lib:/snap/bin WSLENV= _=/usr/bin/env 3. "It looks like the bash does not inherit the environment variables from the kernel. You can test that by setting variables yourself with os.environ in a standard cell and using echo or env in the %%bash cell. You can also compare the behavior between %%bash and %%python." I actually don't know what this means and how to do it(: Thanks for your patience, Vishal

…

On Wed, Oct 4, 2023 at 10:05 AM Vaclav Petras ***@***.***> wrote: The PDF is helpful. It shows that bash itself works. Can you run the env command in your %%bash cell (clean up the output before copy-pasting it here in case there is something private) and try some of the exclamation lines. It looks like the bash does not inherit the environment variables from the kernel. You can test that by setting variables yourself with os.environ in a standard cell and using echo or env in the %%bash cell. You can also compare the behavior between %%bash and %%python. — Reply to this email directly, view it on GitHub <#3195 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJSSRTTMMXLNVZY4FS67WS3X5WJPDAVCNFSM6AAAAAA5RK2QUGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TCOBZGU2TI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

1 reply

wenzeslaus Oct 4, 2023
Maintainer

Python code and exclamation mark lines work, so the GRASS session was set up correctly in the notebook.

It looks like the bash does not inherit the environment variables from the kernel.

If the env output is complete, it still looks like the Bash running from %%bash does not inherit any of the environment variables.

For example, when I run this in Ubuntu command line, I get (env runs inside GRASS session there):

$ grass --tmp-location XY --exec env
...
PATH={path_to_grass}/bin:{path_to_grass}/scripts:/home/vpetras/.grass8/addons/bin:/home/vpetras/.grass8/addons/scripts:/home/vpetras/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin:/home/vpetras/.local/bin:/home/vpetras/.local/bin
...
GISBASE=/home/vpetras/Projects/grass/code/grass/dist.x86_64-pc-linux-gnu
LANGUAGE=en_US
...
LC_NUMERIC=C
GRASS_VERSION=8.4.0dev
GIS_LOCK=4096359
TMPDIR=/tmp/grass8-vpetras-4096359
GISRC=/tmp/grass8-vpetras-4096359/gisrc
GRASS_ADDON_BASE=/home/vpetras/.grass8/addons
PYTHONPATH=/home/vpetras/Projects/grass/code/grass/dist.x86_64-pc-linux-gnu/etc/python
LD_LIBRARY_PATH=/home/vpetras/Projects/grass/code/grass/dist.x86_64-pc-linux-gnu/lib
...

You will get something similar if you do print(os.environ) in a standard Python cell.

You can test that by setting variables yourself with os.environ in a standard cell and using echo or env in the %%bash cell. You can also compare the behavior between %%bash and %%python.

You need a set of cells similar to this (not tested):

os.environ["TEST_VAR_1"] = "Hello"

print("Directly from notebook:", os.environ["TEST_VAR_1"])

%%bash
echo "From Bash subprocess:" $TEST_VAR_1

%%python
print("From Python subprocess:", os.environ["TEST_VAR_1"])

Notably, you can now always run GRASS tools using --exec (that was not available back in the day):

grass --tmp-location XY --exec g.region -p
grass --tmp-mapset /data/north_carolina --exec analysis.sh
grass /data/north_carolina/new_road --exec v.buffer input=...

where grass is the executable, so on Linux usually just grass. On Windows that may be full path to the .bat executable or not depending on what is on the path. In Bash on Windows, it again depends on the path variable (PATH). May you can also try that in your %%bash cell.

vishalm1975 · 2023-10-11T17:12:40Z

vishalm1975
Oct 11, 2023
Author

Thanks Vaclav, I tested what you suggested, and it appears your're right that the bash running through %%bash is not inheriting the env variables. Any ideas how I can fix that? Vishal

…

On Wed, Oct 4, 2023 at 10:05 AM Vaclav Petras ***@***.***> wrote: The PDF is helpful. It shows that bash itself works. Can you run the env command in your %%bash cell (clean up the output before copy-pasting it here in case there is something private) and try some of the exclamation lines. It looks like the bash does not inherit the environment variables from the kernel. You can test that by setting variables yourself with os.environ in a standard cell and using echo or env in the %%bash cell. You can also compare the behavior between %%bash and %%python. — Reply to this email directly, view it on GitHub <#3195 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJSSRTTMMXLNVZY4FS67WS3X5WJPDAVCNFSM6AAAAAA5RK2QUGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TCOBZGU2TI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

1 reply

wenzeslaus Oct 12, 2023
Maintainer

I tried to search for that online before, but didn't get anything. Maybe open an issue for the ipykernel?

As a workaround, you can either use exclamation mark syntax:

!g.list ...

The --exec parameter:

grass ... --exec ...

Or write a Bash script file with %%writefile and then execute that with --exec in a following cell.

vishalm1975 · 2023-10-12T03:36:09Z

vishalm1975
Oct 12, 2023
Author

Thanks Vaclav I appreciate your help. Wow, I find it very sad that I just can't work with grass much anymore because of these issues. There's something fundamental here that needs to be resolved...one should not have to be a developer to use the fantastic tools. Best wishes, Vishal

…

On Wed, Oct 11, 2023, 6:44 PM Vaclav Petras ***@***.***> wrote: I tried to search for that online before, but didn't get anything. Maybe open an issue for the ipykernel? As a workaround, you can either use exclamation mark syntax: !g.list ... The --exec parameter: grass ... --exec ... Or write a Bash script file with %%writefile and then execute that with --exec in a following cell. — Reply to this email directly, view it on GitHub <#3195 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJSSRTSZJJFDQWYHCAYKJ5LX65DO7ANCNFSM6AAAAAA5RK2QUE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GRASS on Windows with Jupyter Notebooks unable to use ##bash #3195

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

GRASS on Windows with Jupyter Notebooks unable to use ##bash #3195

vishalm1975 Oct 2, 2023

Replies: 4 comments · 4 replies

wenzeslaus Oct 3, 2023 Maintainer

vishalm1975 Oct 3, 2023 Author

wenzeslaus Oct 4, 2023 Maintainer

vishalm1975 Oct 4, 2023 Author

wenzeslaus Oct 4, 2023 Maintainer

vishalm1975 Oct 11, 2023 Author

wenzeslaus Oct 12, 2023 Maintainer

vishalm1975 Oct 12, 2023 Author

vishalm1975
Oct 2, 2023

Replies: 4 comments 4 replies

wenzeslaus
Oct 3, 2023
Maintainer

vishalm1975 Oct 3, 2023
Author

wenzeslaus Oct 4, 2023
Maintainer

vishalm1975
Oct 4, 2023
Author

wenzeslaus Oct 4, 2023
Maintainer

vishalm1975
Oct 11, 2023
Author

wenzeslaus Oct 12, 2023
Maintainer

vishalm1975
Oct 12, 2023
Author