1.19.0rc2: test_full_reimport fails with "RuntimeError: implement_array_function method already has a docstring"

[sandrotosi]

Hello,
this has started appearing with rc2 (maybe also rc1, but it didnt happen in 1.18.4):

=================================== FAILURES ===================================
______________________________ test_full_reimport ______________________________

    def test_full_reimport():
        """At the time of writing this, it is *not* truly supported, but
        apparently enough users rely on it, for it to be an annoying change
        when it started failing previously.
        """
        # Test within a new process, to ensure that we do not mess with the
        # global state during the test run (could lead to cryptic test failures).
        # This is generally unsafe, especially, since we also reload the C-modules.
        code = textwrap.dedent(r"""
            import sys
            import numpy as np
    
            for k in list(sys.modules.keys()):
                if "numpy" in k:
                    del sys.modules[k]
    
            import numpy as np
            """)
        p = subprocess.run([sys.executable, '-c', code])
>       assert p.returncode == 0
E       assert 1 == 0
E         -1
E         +0

code       = ('\n'
 'import sys\n'
 'import numpy as np\n'
 '\n'
 'for k in list(sys.modules.keys()):\n'
 '    if "numpy" in k:\n'
 '        del sys.modules[k]\n'
 '\n'
 'import numpy as np\n')
p          = CompletedProcess(args=['/usr/bin/python3.8', '-c', '\nimport sys\nimport numpy as np\n\nfor k in list(sys.modules.keys()):\n    if "numpy" in k:\n        del sys.modules[k]\n\nimport numpy as np\n'], returncode=1)

tmp/usr/lib/python3/dist-packages/numpy/tests/test_reloading.py:57: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 9, in <module>
  File "/usr/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 16, in <module>
    add_docstring(
RuntimeError: implement_array_function method already has a docstring

this is running the test suite in our build env, which is a bare minimum chroot with only few packages installed; full log of an example is here: https://buildd.debian.org/status/fetch.php?pkg=numpy&arch=amd64&ver=1%3A1.19.0%7Erc2-1&stamp=1591237245&raw=0

similar issues (not sure how much relation there is): #14012, #14384, #15563

[charris]

I wonder why this isn't seen in the other testing environments, there has got to be some difference.

[charris]

What happens if you use the software versions in test_requirements.txt?

[charris]

@seberg Thoughts?

[seberg]

Thanks for the ping. This is weird because the error should read already has a different docstring (note the additional "different"). Any chance this is running the previous RCs code, but with the new tests? I admit that seems unlikely, but...

[seberg]

Hmm, or the test being called in a sub-process for some reason picks up a different NumPy version? And in that case, do we have to detect that?

[charris]

That's an interesting, but disturbing, thought :)

[charris]

Maybe running in a virtual environment would fix that?

[sandrotosi]

Thanks for the ping. This is weird because the error should read already has a different docstring (note the additional "different"). Any chance this is running the previous RCs code, but with the new tests? I admit that seems unlikely, but...

that's very likely what's happening! to build numpy, we install its build dependencies, some of which already depends on the packaged numpy, so that gets installed as well; f.e. from the build log in the original post:

Setting up python3-numpy (1:1.18.4-1) ...

so that's the system-wide available version of numpy.

Then we start building the new version, and we do try hard to run tests/docs/examples against the being-built version, but running a subprocess with python3 -c "import numpy" will very likely hit the system-wide version of numpy, so 1.18.4

is test_full_reimport relatively new? not sure what's the right approach here, maybe i can just skip it in the debian build?

[seberg]

@sethtroisi, yes we squeezed it into the 1.19 release, because it was annoying some users (even if the whole patch is at this time a bit dangerous).

Hmmm, I am considering passing in the actual numpy version and skip the test when they do not match?

[seberg]

Or wait. Are the environment variables not inherited here? So that e.g. PYTHONPATH is set for the test run, but not in the subprocess. If so, can we just ensure that the subprocess inherits all environment variables?

[charris]

You could try numpy's runtests.py, which tests in a virtual environment.

[seberg]

Ah, subprocess does inherit environment, so I suppose the issue is indeed the setup entirely running the previous NumPy?

[sethtroisi]

@sethtroisi, yes we squeezed it into the 1.19 release, because it was annoying some users (even if the whole patch is at this time a bit dangerous).

Hmmm, I am considering passing in the actual numpy version and skip the test when they do not match?

@sandrotosi (seberg) meant to ping you here

[sandrotosi]

Ah, subprocess does inherit environment, so I suppose the issue is indeed the setup entirely running the previous NumPy?

the way we're running tests is by injecting the newly built location as the first element in sys.path, so running python3 -c "import numpy; etc as in the test will not have that path set, and will use the system-wide numpy installation

[seberg]

@sandrotosi but maybe than we just should do that as well, I assume you mean -c "import sys; sys.path.insert(-1, current_path)"?

[sandrotosi]

it's python$$v -c "import sys ; sys.path.insert(0, './tmp/usr/lib/python3/dist-packages/') ; import numpy; numpy.test(verbose=5)" ; (with $v being the version of python we're currently testing). this is very specific to how debian builds its packages (if you want to look at the details, they are here)

i'm not sure that's a valid general approach for numpy upstream (f.e. what would happen if you install numpy and then just run import numpy as np; np.test()?)

[seberg]

Would instead running it using PYTHONPATH=./tmp/usr/lib/python3/dist-packages/:$PYTHONPATH be fine for you?

I thought maybe we can add the path of np.__file__, just like you are doing it here. But, I am also happy to just go back to the old skipif idea, which at least doesn't need messing about with paths. These are not super important tests... I am surprised we do not have more tests like that, but I guess it is possible all of those simply pass with older numpy versions.

[seberg]

Or maybe you just can switch directories to anywhere else?

[sandrotosi]

I remember having a bit of a hard time figuring out the right way to run tests during the build process, since numpy tries really hard not to let you import it from the source dir :)

we're already skipping a couple tests, test_compile1,2 so i guess i can add this one, as it probably has more value during CI than at build time.

thanks for your help on this, going to close this issue now and address it in debian!

[seberg]

Blacklisting this one is definitely fine. I wish we could help you more though. I would think that simply change of directory (e.g. to a temporary one) would be fine, but I guess there are probably some subtleties I am missing...