Reloader, python -m, and sys.path

[SimonSapin]

See Kozea/WeasyPrint#133

What’s going on is:

• python -m weasyprint.navigator is run
• [sys.executable] + sys.argv is ['…/python', '…/weasyprint/navigator.py']
• The reloader spawns a subprocess that way
• The child process does not have -m, so Python add’s the .py file’s parent directory to sys.path
• Werkzeug tries to import stdlib’s html.entities
• This incorrectly imports weasyprint/html.py
• Stuff breaks

Nobody here is doing anything obviously wrong. How do you suggest dealing with this situation? What do you think of having …/weasyprint/navigator.py (which is expected to be mostly used with python -m) remove its parent directory from sys.path if it’s there?

[untitaker]

I don't think it would not harm if Werkzeug spawned the subprocess with actually the same arguments, using __loader__.fullname in Python 2 and __loader__.name in Python 3 if existent.

[SimonSapin]

@untitaker, I don’t understand what you mean. What’s __loader__?

Can we detect in Python if python -m somemodule was used or not?

[untitaker]

I camne across a StackOverflow answer that explained how to get the current module name. A simple test shows that in Python 2, __loader__.fullname is available:

print(__loader__.fullname)

The __loader__ global doesn't exist if the Python file is executed without -m, and it also doesn't exist in imported modules.

In Python 3, both the module executed with -m and any other imported modules seem to have a __loader__ global, the fullname attribute is now name. But inside every module, the name attribute has the value of the current module.

So, in Python 2 we can detect if we're running with -m, but not in 3.

[untitaker]

Also the detection is very limited either, so you only can detect inside the directly executed module if python -m was used by checking for the __loader__ variable. Also, this behavior seems sparsely documented, so i am not sure if that's what we want in Werkzeug anyway.

[waynew]

I just came across this bug myself. I expected the -m flag to be present in sys.argv, but it most definitely isn't.

Is there any kind of workaround for this?

[untitaker]

Yes, one is presented in pallets/flask#1246

On 9 July 2015 15:08:30 CEST, Wayne Werner notifications@github.com wrote:

I just came across this bug myself. I expected the -m flag to be
present in sys.argv, but it most definitely isn't.

Is there any kind of workaround for this?

---

Reply to this email directly or view it on GitHub:
#461 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[martijnvermaat]

I have a similar issue, where relative imports fail because the reloader executes the module as a non-package:

$ python -m myapp.entrypoints.website
 * Running on http://127.0.0.1:5000 (Press CTRL+C to quit)
 * Restarting with stat
Traceback (most recent call last):
  File "/tmp/myapp/entrypoints/website.py", line 9, in <module>
    from . import mymodule
ValueError: Attempted relative import in non-package

Using the comments from @untitaker, in Python < 3.3 we can detect if python -m was used (only then is __loader__ defined) and force the reloader to also use it by resetting sys.argv:

try:
    sys.argv = ['-m', __loader__.fullname] + sys.argv[1:]
except NameError:
    pass

But this is quite a hacky workaround (and only for Python < 3.3).

The current implementation in #531 doesn't fix this particular problem and unfortunately I don't see any way how it could.

[untitaker]

I think I tried to fix this with loader once, but it showed vastly
different behavior from 2 to 3.

On Thu, Sep 10, 2015 at 01:26:56PM -0700, Martijn Vermaat wrote:

I have a similar issue, where relative imports fail because the reloader executes the module as a non-package:

$ python -m myapp.entrypoints.website
 * Running on http://127.0.0.1:5000 (Press CTRL+C to quit)
 * Restarting with stat
Traceback (most recent call last):
  File "/tmp/myapp/entrypoints/website.py", line 9, in <module>
    from . import mymodule
ValueError: Attempted relative import in non-package

Using the comments from @untitaker, in Python < 3.3 we can detect if python -m was used (only then is __loader__ defined) and force the reloader to also use it by resetting sys.argv:

try:
    sys.argv = ['-m', __loader__.fullname] + sys.argv[1:]
except NameError:
    pass

But this is quite a hacky workaround (and only for Python < 3.3).

---

Reply to this email directly or view it on GitHub:
#461 (comment)

[asarch]

Sadly,

$ python -m werkzeug.serving weasyprint.navigator:app --reload --debug

• Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
• Restarting with stat
  Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/werkzeug/serving.py", line 45, in
  from ._compat import PY2
  ValueError: Attempted relative import in non-package

[RonnyPfannschmidt]

This bug of python can't be fixed propperly with python, so I suggest to provide a console-script

[thpica]

I managed to get around the issue by doing something like:

PYTHONPATH=$PWD:$PYTHONPATH python -m myapp.entrypoints.website

Edit: realized the solution was just above ⬆️