Numpy can't be used in embedded player 2x in row without restarting

[BluePrintRandom]

https://www.youtube.com/watch?v=aNqfsnGQKRQ

if we import numpy, close engine -> start again -> import numpy we get some error messages

nummpy.zip

[paul-marechal]

Does using module mode help, rather than script mode?

[BluePrintRandom]

nope*

[paul-marechal]

Seems to be a bug with numpy itself.

The BGE makes a backup of the modules before running the embedded player, and restores it at the end. This ensures that modules are not cached for eternity (until you restart Blender itself), and allows your game to always load modules from disk. Else it would always hit the cache and never update modules.

The embedded player does not have this issue since no backup and restore is required.

The approximate call chain leading to the backup of the modules:

file: /source/gameengine/Ketsji/KX_PythonInit.cpp

- initGamePlayerPythonScripting
  - initPySysObjects
    - backupPySysObjects

I guess you can wrap numpy's importation somehow, since the sys object is not entirely backup'ed:

# file: numpy_wrapper.py

import sys

if 'numpy' not in sys.modules and hasattr(sys, '__numpy__'):
    # restore module cache with backup ref
    numpy = sys.modules['numpy'] = sys.__numpy__

else:
    import numpy
    # store a backup ref
    sys.__numpy__ = numpy

And then to use it:

from numpy_wrapper import numpy

[youle31]

I can't reproduce a bug with numpy in last master with this test file:

https://pasteall.org/blend/c18e144cf3194a829ccdc7cce2851f56

(neither in embedded, neither in standalone)

[BluePrintRandom]

Nice!

…

On Thu, Apr 30, 2020, 10:10 AM youle31 ***@***.***> wrote: I can't reproduce a bug with numpy in last master with this test file: https://pasteall.org/blend/c18e144cf3194a829ccdc7cce2851f56 (neither in embedded, neither in standalone) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABW3SWK5C43TAQIA3I2OWATRPGWJFANCNFSM4KMWOBCA> .

[BluePrintRandom]

this is from this build hash

it fails the second time I run your test file @youle31

[BluePrintRandom]

(the first time it made kinda a rainbow on the cube?)
and in standalone

[youle31]

I can't reproduce this bug (even if I restart 50 times, I have no error message). Yes, it is an example from gpu module doc

[lordloki]

I can confirm it.

It seems that is an issue of numpy itself (numpy/numpy#14384)
A mitigation as pointer by an user in the numpy github tracker is to downgrading numpy to numpy==1.15.4
I'm going to see if it is possible this solution

[lordloki]

It works!! but then we have a new issue with the version 1.15.4. We have the following error

and then we don't have audio. We can not solve this because _multiarray_umath needs numpy 1.16 and the numpy reload needs version 1.15.4

We will have to live with this issue.

[BluePrintRandom]

Ok - since we can use it in standalone it's not that big of a deal,

maybe we can think of a fix long term and come back to this?

I will close for now,

[paul-marechal]

The issue comes from the fact that numpy does monkey patching on module initialization, and the BGE re-initialize numpy in a global Python context where the monkey patch cannot be applied twice. You can find the workaround code here:

# file: numpy_wrapper.py

import sys

if 'numpy' not in sys.modules and hasattr(sys, '__numpy__'):
    # restore module cache with backup ref
    numpy = sys.modules['numpy'] = sys.__numpy__

else:
    import numpy
    # store a backup ref
    sys.__numpy__ = numpy

And then to use it:

from numpy_wrapper import numpy

[lordloki]

Thanks. I will include in the release notes.

[paul-marechal]

Just tried to import numpy on 0.2.5 and I don't get the crash... But when it fails with the ... already has a docstring my workaround should apply.

[BluePrintRandom]

no crash for me either - when I say "without restarting" I mean shutting down blender and opening it again - the file works again =

…

On Thu, Apr 30, 2020 at 5:50 PM Paul Maréchal ***@***.***> wrote: Just tried to import numpy on 0.2.5 and I don't get the crash... But when it fails with the ... already has a docstring my workaround should apply. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABW3SWJNR7D6VLKAVWCAOSTRPIMGVANCNFSM4KMWOBCA> .

[BluePrintRandom]

Is there a way to use your workaround if in embedded automatically
?

[youle31]

it's weird that you have this error and not me, maybe it is because of the way we build blender...

EDIT: I have with_PYTHON_INSTALL_NUMPY enabled by default in my cmake. I build with cmake 3.15.3

[lordloki]

I also have WITH_PYTHON_INSTALL_NUMPY enabled and my cmake is 3.15.3.
Very weird