tcl is not included to cpython python 3.11

[seonglae]

Steps to Reproduce

rye pin cpython@3.11
rye init
rye sync
python

from tkinter import *
Tk()

Expected Result

no error

Actual Result

Python 3.11.1 (main, Jan 16 2023, 16:02:03) [Clang 15.0.7 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tkinter import *
>>> Tk()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_tkinter.TclError: Can't find a usable init.tcl in the following directories: 

Version Info

rye --version
error: unexpected argument found

Stacktrace

No response

[mitsuhiko]

This is sort of expected at the moment. You need to tell TCL where it can find the init.tcl file. I am actually surprised tkinter is still being used which is why I did not bother looking into improving that experience yet.

[mitsuhiko]

For now run this before importing tkinter:

import os
import sys
os.environ["TCL_LIBRARY"] = sys.base_prefix + "/lib/tcl8.6"
os.environ["TK_LIBRARY"] = sys.base_prefix + "/lib/tk8.6"

The only way in which I can automate this is to set those environment variables. Need to decide if that's a good idea still.

[mitsuhiko]

In theory rye could throw a .pth file like this into the virtualenv to make it work:

import os, sys; os.environ.setdefault("TCL_LIBRARY", sys.base_prefix + "/lib/tcl8.6"); os.environ.setdefault("TK_LIBRARY", sys.base_prefix + "/lib/tk8.6")

[seonglae]

For now run this before importing tkinter:

import os
import sys
os.environ["TCL_LIBRARY"] = sys.base_prefix + "/lib/tcl8.6"
os.environ["TK_LIBRARY"] = sys.base_prefix + "/lib/tk8.6"

The only way in which I can automate this is to set those environment variables. Need to decide if that's a good idea still.

This solution works well. Thank you for your help

In other way, rm -r .venv && python -m venv .venv && pip install -r requirements-dev.lock also worked if don't want to edit code

[mitsuhiko]

@seonglae that's likely because whatever python binary is picked up there has some hacks in place to pick up the system tcl version. In general the issue is that Python never really found a good solution to initialize tcl and issues with virtualenvs and TCL require custom hacks in general.

• Tkinter - fails in a virtual env pypa/virtualenv#56
• ensure tcl works within virtualenv pypa/virtualenv#93

There are various things that can be done, but it's unclear what is the least hacky way to accomplish that.

[seonglae]

Right. I used global python binary for above solution. Hopefully others will not have to use TK. Thank you for more details.

[mitsuhiko]

This will work out of the box in 0.4.0.

[EthanRosenthal]

For now run this before importing tkinter:

import os
import sys
os.environ["TCL_LIBRARY"] = sys.base_prefix + "/lib/tcl8.6"
os.environ["TK_LIBRARY"] = sys.base_prefix + "/lib/tk8.6"

The only way in which I can automate this is to set those environment variables. Need to decide if that's a good idea still.

I'm trying to use rye with matplotlib and the tkagg background, and this hack only seems to work for me if I use rye shell. Do you have any ideas on how to get this to work if I don't want to spawn a shell?

[mitsuhiko]

@EthanRosenthal this code runs with and without shell. can you provide an example where it fails?

[EthanRosenthal]

Yeah, here's code that fails without the shell

rye init my-project
cd my-project
rye add matplotlib 
rye sync 
echo 'import os
import sys
os.environ["TCL_LIBRARY"] = sys.base_prefix + "/lib/tcl8.6"
os.environ["TK_LIBRARY"] = sys.base_prefix + "/lib/tk8.6"
import matplotlib.pyplot as plt
plt.switch_backend("tkagg")
plt.figure()
plt.show()
' > test_mpl.py
python test_mpl.py

With stacktrace:

Traceback (most recent call last):
  File "/home/er/tmp/my-project/test_mpl.py", line 6, in <module>
    plt.switch_backend("tkagg")
  File "/home/er/tmp/my-project/.venv/lib/python3.11/site-packages/matplotlib/pyplot.py", line 271, in switch_backend
    backend_mod = importlib.import_module(
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/er/.rye/py/cpython@3.11.5/install/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/er/tmp/my-project/.venv/lib/python3.11/site-packages/matplotlib/backends/backend_tkagg.py", line 1, in <module>
    from . import _backend_tk
  File "/home/er/tmp/my-project/.venv/lib/python3.11/site-packages/matplotlib/backends/_backend_tk.py", line 24, in <module>
    from . import _tkagg
ImportError: module '_tkinter' has no attribute '__file__'

If you spawn a shell before running test, then a plot figure will successfully be shown:

rye init my-project
cd my-project
rye add matplotlib 
rye sync 
echo 'import os
import sys
os.environ["TCL_LIBRARY"] = sys.base_prefix + "/lib/tcl8.6"
os.environ["TK_LIBRARY"] = sys.base_prefix + "/lib/tk8.6"
import matplotlib.pyplot as plt
plt.switch_backend("tkagg")
plt.figure()
plt.show()
' > test_mpl.py
rye shell
python test_mpl.py

In case it's relevant, I'm on Ubuntu 22.04. My computer is setup to use pyenv and pyenv-virtualenv for managing python versions and virtual environments, respectively. rye has been working well for everything else except for this matplotlib/tkinter issue.

[mitsuhiko]

It sounds to me like the Python running there might not actually be the rye python executable. The .pth file has nothing to do with if rye shell is used or not. Can you run which python?

[EthanRosenthal]

which python gives the rye shim:

/home/er/.rye/shims/python

here's rye show, in case that's helpful

project: my-project
path: /home/er/tmp/my-project
venv: /home/er/tmp/my-project/.venv
target python: 3.8
venv python: cpython@3.11.5

I wonder if this is due to a weird interaction with pyenv 🤔

[EthanRosenthal]

Ah, wait, you're totally right. The above code was from when I ran which python outside of rye shell. When I run which python within rye shell, I get

/home/er/.pyenv/shims/python

Back to my original goal which is to be able to use matplotlib with rye, any idea why the environment variable hack didn't work for me?