Rendering Rmd with matplotlib with reticulate in IDE seems to required tcl tk

[cderv]

System details

RStudio Edition : Desktop [Open Source]
RStudio Version : 2023.12.0.201
OS Version      : Windows 11 x64 (build 22621)
R Version       : R version 4.3.1 (2023-06-16 ucrt)

> packageVersion("reticulate")
[1] ‘1.34.0.9000’
> packageVersion("rmarkdown")
[1] ‘2.25’

Steps to reproduce the problem

This is a simple Rmarkdown file

---
title: "matplotlib test"
output: html_document
---

```{r}
#reticulate::virtualenv_create("test")
#reticulate::virtualenv_install("test", "matplotlib")
reticulate::use_virtualenv("test")
```


```{python}
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.show()
```

I used a new virtual environment to try isolate the issue.

• Render this with rmarkdown::render() in R console and there is no issue
• Render this with Knit button and there is an issue (see below)

Opening to share the error and see if this is really a IDE issue or not (as explained below, no problem when rendering on its own)

Describe the problem in detail

This is the issue

Error in `py_call_impl()`:
! _tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    C:/Users/chris/PYENV~1/PYENV-~2/versions/311~1.1/lib/tcl8.6 C:/PROGRA~1/R/R-43~1.1/bin/lib/tcl8.6 C:/PROGRA~1/R/R-43~1.1/lib/tcl8.6 C:/PROGRA~1/R/R-43~1.1/bin/library C:/PROGRA~1/R/R-43~1.1/library C:/PROGRA~1/R/R-43~1.1/tcl8.6.12/library C:/PROGRA~1/R/tcl8.6.12/library



This probably means that Tcl wasn't installed properly.

Run `reticulate::py_last_error()` for details.
Backtrace:
  1. rmarkdown::render("C:/Users/chris/Documents/test.Rmd", encoding = "UTF-8")
  2. knitr::knit(knit_input, knit_output, envir = envir, quiet = quiet)
  3. knitr:::process_file(text, output)
  7. knitr:::process_group.block(group)
  8. knitr:::call_block(x)
     ...
 12. knitr (local) engine(options)
 13. reticulate::eng_python(options)
 14. reticulate:::py_compile_eval(snippet, compile_mode)
 17. builtins$eval(compiled, globals, locals)
 18. reticulate:::py_call_impl(callable, call_args$unnamed, call_args$named)

It seems the tk backend is tried to be used. I even change Graphics settings to use AGG. But it does not work.

reticulate is supposed to set the agg backend when inside RStudio desktop
https://github.com/rstudio/reticulate/blob/5d0a9a52286a173ac8868c1d548b8715247867ae/R/knitr-engine.R#L484-L501
Probably why no error when render is called in R console

Adding the backend switch in the chunk makes the knit button works with no problem

```{python}
import matplotlib.pyplot as plt
plt.switch_backend('agg')
plt.plot([1,2,3,4])
plt.show()
```

Describe the behavior you expected

I would expect knitting in IDE works the same as calling rmarkdown::render() with no error. Especially as reticulate seems to do the work of setting the backend.

I am sure this worked before, so maybe this is something that changed in recent version or related to some settings.

• I have read the guide for submitting good bug reports.
• I have installed the latest version of RStudio, and confirmed that the issue still persists.
• If I am reporting an RStudio crash, I have included a diagnostics report.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

[matulad]

I've encountered the same problem while rendering Quarto document with both r and python cells in VSCode.

[L3vL]

I've encountered the same problem while rendering Quarto document with both r and python cells in VSCode.

Same issue here. Did you find any solution so far?

[daxkellie]

I managed to find a very manual solution to this issue (thanks in part to this thread)

In my case, I was trying to render a Quarto document with both R and Python cells in R Studio and hitting the exact same error. It seems that for whatever reason, the process to find the Tcl folder doesn't look in the correct place of where it is installed outside of your virtual environment, and so rendering returns the error you saw that it Can't find a usable init.tcl in the following directories: followed by a list of those directories.

The trick is to find the tcl8.6 folder on your computer (in my case, it was in C:/Users/[my-username]/Anaconda3/tcl), copy the tcl8.6 folder and paste it in one of directories that is being looked up during the rendering process.

In your case (according to your error message), the first place the tcl8.6 is being looked for is C:/Users/chris/PYENV~1/PYENV-~2/versions/311~1.1/lib/tcl8.6, but it must not be there. Copy the tcl8.6 folder wherever it currently is, and paste it inside C:/Users/chris/PYENV~1/PYENV-~2/versions/311~1.1/lib. You will also need to copy/paste the tk8.6 folder (found inside the tcl folder) into the tcl8.6 folder you've created. This should do it!

In case you don't know where your Tcl folder is located, I managed to find mine by trying to install Tcl in the Anaconda Prompt Terminal by running:
python -m pip install Tcl

This returned:
Requirement already satisfied: Tcl in c:\users\[my-username]\anaconda3\lib\site-packages (0.2)

Hopefully this issue is fixed soon so we can avoid this process!

[cderv]

Thanks for sharing workaround !

I know where my tcl8.6 is in the python installation - and settign TCL_LIBRARY env var to it, will indeed help make this works (no need to copy directory in that case)

---
title: "matplotlib test"
output: html_document
---

```{r}
# Using tcl available in the Python installation (outside virtual env)
Sys.setenv(TCL_LIBRARY = file.path(dirname(system("pyenv which python", intern = TRUE)), "tcl", "tcl8.6"))
```

```{r}
#reticulate::virtualenv_create("test")
#reticulate::virtualenv_install("test", "matplotlib")
reticulate::use_virtualenv("test")
```


```{python}
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.show()
```

Problably adjusting directory is long term solution for this virtual env

Though, I believe the IDE should try to use AGG like reticulate packages does. I don't yet understand why the difference between rendering in IDE, and rendering using R console... 🤷

[christophscheuch]

Interestingly, the tcl issue does not arise on Mac ("arm64"), but it does on Windows ("x86-64")...

If you work with reticulate and renv, then the following might be interesting for you. Since I work on the same project on two machines (Windows and Mac), I added this to my .Rprofile:

if (Sys.info()["machine"] == "x86-64") {
  user_name <- Sys.info()["user"]
  tcl_path <- paste0("C:/Users/", user_name, "/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.10.11/tcl/tcl8.6")
  Sys.setenv(TCL_LIBRARY = tcl_path)
}

I did not use system("which python", intern = TRUE) because this returns the path to the python environment in renv, which has no tcl library. Hence, I hard coded the Python version in the path above (I always use the same Python version anyway because it is the most recent one that supports everything I'm working on on both OS so far).

Note that I use reticulate and renv on both machines. If you also want to set it up that way, these steps have worked for me:

1. install.packages("renv")
2. renv::activate()
3. install.packages("reticulate")
4. reticulate::install_python(version="3.10.11", force = TRUE)
5. Tell renv to use python:

• On Windows: renv::use_python(paste0("C:/Users/", Sys.info()["user"], "/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.10.11/python.exe"))
• On Mac: renv::use_python("~/.pyenv/versions/3.10.11/bin/python")

6. renv::snapshot()

Feel free to check out an example project where I use this here.

[kevinushey]

I'm trying to test this out now, and I see that agg is being used:

```{python}
import matplotlib.pyplot as plt
print(plt.get_backend())
```

And that seems to indicate agg is being used:

>>> print(plt.get_backend())
agg

Note that the RStudio hook that tries to force matplotlib to use "agg" requires the png package to be installed, e.g.

rstudio/src/cpp/session/modules/SessionReticulate.R

Lines 178 to 184 in 10551e8

	# install matplotlib hook if available
	canInstallHooks <-
	requireNamespace("png", quietly = TRUE) &&
	reticulate::py_module_available("matplotlib")
	if (!canInstallHooks)
	return()

@cderv if you're still hitting this could we take some time to debug together?

[cderv]

This is still happening for me and agg is not set, while png is available. 🤔

This is with 2024.04.0-daily+581 on Windows

We can definitely look into debugging this together

This is only with Knit button and rendering happens in render pane.

Using rmarkdown::render() in the R console, render the document and agg is set.

If I look at where this tkAgg is from I am finding this

rstudio/src/cpp/session/modules/SessionReticulate.R

Lines 113 to 117 in 10551e8

	# try to default to the tkAgg backend (note that we'll
	# force a vanilla 'agg' backend when required otherwise)
	engine <- tolower(Sys.getenv("MPLENGINE"))
	if (engine %in% c("", "qt5agg"))
	Sys.setenv(MPLENGINE = "tkAgg")

So the envar is set, if unset. and this is not overridden by agg it seems 🤔

[kevinushey]

Ah, for some reason I missed that this issue occurs specifically when using the Knit button. I tested when executing code directly in the IDE, and that's where we more forcefully set the backend to agg.

I wonder if the code in reticulate above is wrong? It looks like the environment used to configure a default backend with matplotlib is normally called MPLBACKEND, not MPLENGINE.

https://matplotlib.org/stable/users/explain/figure/backends.html

If so, that would imply the fix should happen in reticulate...

[cderv]

I wonder if the code in reticulate above is wrong? It looks like the environment used to configure a default backend with matplotlib is normally called MPLBACKEND, not MPLENGINE.

Which code of reticulate are you referring to ?

It looks like the environment used to configure a default backend with matplotlib is normally called MPLBACKEND, not MPLENGINE.

It seems indeed that this is not working, and the tkAgg value comes from default matplotlib

> RScript -e "Sys.setenv(MPLENGINE = 'agg'); reticulate::py_run_string('import matplotlib.pyplot as plt;print(plt.get_backend())')"
++ Activating rlang global_entrace

TkAgg
> RScript -e "Sys.setenv(MPLBACKEND = 'agg'); reticulate::py_run_string('import matplotlib.pyplot as plt;print(plt.get_backend())')"
++ Activating rlang global_entrace

agg

I does get the same value in Python directly

Python 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.pyplot as plt
>>> print(plt.get_backend())
TkAgg

Following hint at https://matplotlib.org/stable/users/explain/figure/backends.html#selecting-a-backend I see

Without a backend explicitly set, Matplotlib automatically detects a usable backend based on what is available on your system and on whether a GUI event loop is already running. The first usable backend in the following list is selected: MacOSX, QtAgg, GTK4Agg, Gtk3Agg, TkAgg, WxAgg, Agg.

So I believe I get TkAgg selected this way, and that somehow RStudio does not change it

rstudio/src/cpp/session/modules/SessionReticulate.R

Lines 186 to 198 in 10551e8

	matplotlib <- reticulate::import("matplotlib", convert = TRUE)
	# force the "Agg" backend (this is necessary as other backends may
	# fail with RStudio if requisite libraries are not available)
	backend <- matplotlib$get_backend()
	if (!identical(tolower(backend), "agg"))
	{
	sys <- reticulate::import("sys", convert = TRUE)
	if ("matplotlib.backends" %in% names(sys$modules))
	matplotlib$pyplot$switch_backend("agg")
	else
	matplotlib$use("agg", warn = FALSE, force = TRUE)
	}

While when using reticulate inside RStudio it seems to work at
https://github.com/rstudio/reticulate/blob/5d0a9a52286a173ac8868c1d548b8715247867ae/R/knitr-engine.R#L484-L501

anyhow, happy to go live with you to try debug further

[kevinushey]

Maybe the code block at https://github.com/rstudio/reticulate/blob/5d0a9a52286a173ac8868c1d548b8715247867ae/R/knitr-engine.R#L484-L501 needs to be run more aggressively? Maybe force its use in non-interactive sessions? I can't think of a reason why a user would want a non-agg backend in that case...

We could have an environment variable like RETICULATE_MPLBACKEND as an override if it were needed.

[cderv]

Maybe the code block at rstudio/reticulate@5d0a9a5/R/knitr-engine.R#L484-L501 needs to be run more aggressively? Maybe force its use in non-interactive sessions?

I do think it is not run in background session in fact.

> callr::r(function() { reticulate:::is_rstudio_desktop() })
[1] FALSE

So when doing Knit button, it runs render in background, and it doesn't apply the agg setting.

Running in console this works

rmarkdown::render('test.Rmd')

but in background is does not change tkAgg to agg

callr::r(function() { rmarkdown::render('test.Rmd'); })

and that would be why it doesn't change.

I would have thought the rstudio setting would apply

rstudio/src/cpp/session/modules/SessionReticulate.R

Lines 176 to 198 in 10551e8

	.rs.addFunction("reticulate.matplotlib.onLoaded", function()
	{
	# install matplotlib hook if available
	canInstallHooks <-
	requireNamespace("png", quietly = TRUE) &&
	reticulate::py_module_available("matplotlib")
	if (!canInstallHooks)
	return()
	matplotlib <- reticulate::import("matplotlib", convert = TRUE)
	# force the "Agg" backend (this is necessary as other backends may
	# fail with RStudio if requisite libraries are not available)
	backend <- matplotlib$get_backend()
	if (!identical(tolower(backend), "agg"))
	{
	sys <- reticulate::import("sys", convert = TRUE)
	if ("matplotlib.backends" %in% names(sys$modules))
	matplotlib$pyplot$switch_backend("agg")
	else
	matplotlib$use("agg", warn = FALSE, force = TRUE)
	}

but the hook does not apply it seems.

So yes applying more aggressively seems interesting - or just using another way to test that we are running from RStudio

I can't think of a reason why a user would want a non-agg backend in that case...

Yes I agree.

[kevinushey]

QA note: testing requires the development version of reticulate, so please do:

remotes::install_github("rstudio/reticulate")

and restart the R session before attempting to verify.

[ronblum]

Verified in RStudio Desktop 2024.04.0-daily+668on Windows 11. Checked both knitting and rendering from the console. In both cases, the plots display correctly, and print(plt.get_backend()) returns agg.