Instantiating multiprocessing.SimpleQueue() from an imported package fails unless SimpleQueue() is instantiated inside Twisted application first

[michelcrypt4d4mus]

Describe the incorrect behavior you saw

See discussion. Ran into a weird situation where I was importing a package macos_notifications. That package in turn instantiates a multiprocessing.SimpleQueue object - or at least it tries to. Just the simple instantiation call fails with a bad value(s) in fds_to_keep error...

...unless, I discovered, I instantiate a SimpleQueue object inside my Twisted application somewhere, anywhere. Then there's no error and things more or less work fine albeit with warnings about uncollected resources when the Twisted server exits:

resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown

Describe how to cause this behavior

See exact steps below

Inside of a twisted.application (specifically opencanary) running some twisted servers (both internet.TCPServer and internet.UDPServer, roughly 10 at the same time) do:

    from mac_notifications import client

    client.create_notification(
        title="Meeting starts now!",
        subtitle="Team Standup",
        action_button_str="Join zoom meeting",
        action_callback=partial(join_zoom_meeting, conf_number=zoom_conf_number)
    )

I would presume the same error might arise from other packages that choose to instantiate a SimpleQueue() but macos_notifications is the only one I tested.

Stack trace

2024-05-19T15:34:59-0400 [opencanary.modules.ssh.HoneyPotSSHFactory] Unhandled Error
	Traceback (most recent call last):
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/twisted/python/log.py", line 80, in callWithContext
	    return context.call({ILogContext: newCtx}, func, *args, **kw)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
	    return func(*args, **kw)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/twisted/internet/selectreactor.py", line 148, in _doReadOrWrite
	    why = getattr(selectable, method)()
	--- <exception caught here> ---
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/twisted/internet/tcp.py", line 1410, in doRead
	    protocol.makeConnection(transport)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/twisted/internet/protocol.py", line 509, in makeConnection
	    self.connectionMade()
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/opencanary/modules/ssh.py", line 156, in connectionMade
	    log(logdata, transport=self.transport, logtype=logtype)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/opencanary/modules/__init__.py", line 100, in log
	    client.create_notification(**notification_kwargs)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/mac_notifications/client.py", line 66, in create_notification
	    return get_notification_manager().create_notification(notification_config)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/mac_notifications/client.py", line 21, in get_notification_manager
	    return NotificationManager()
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/mac_notifications/singleton.py", line 17, in __call__
	    instance = super().__call__(*args, **kwargs)
	  File "/Users/uzor/workspace/opencanary/env/lib/python3.12/site-packages/mac_notifications/manager.py", line 51, in __init__
	    self._callback_queue: SimpleQueue = SimpleQueue()
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/context.py", line 113, in SimpleQueue
	    return SimpleQueue(ctx=self.get_context())
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/queues.py", line 363, in __init__
	    self._rlock = ctx.Lock()
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/context.py", line 68, in Lock
	    return Lock(ctx=self.get_context())
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/synchronize.py", line 169, in __init__
	    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/synchronize.py", line 80, in __init__
	    register(self._semlock.name, "semaphore")
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 174, in register
	    self._send('REGISTER', name, rtype)
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 182, in _send
	    self.ensure_running()
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 148, in ensure_running
	    pid = util.spawnv_passfds(exe, args, fds_to_pass)
	  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/util.py", line 456, in spawnv_passfds
	    return _posixsubprocess.fork_exec(
	builtins.ValueError: bad value(s) in fds_to_keep

Describe the correct behavior you'd like to see

I shouldn't need to instantiate SimpleQueue() to get SimpleQueue() to work in packages used by my twisted app.

Testing environment

• Operating System and Version; paste the output of these commands:

ProductName:		macOS
ProductVersion:		13.6.4
BuildVersion:		22G513

• Twisted version: 24.3.0

pip freeze:

attrs==23.2.0
Automat==22.10.0
bcrypt==3.1.7
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
constantly==23.10.4
cryptography==38.0.1
hpfeeds==3.0.0
hyperlink==21.0.0
idna==3.7
incremental==22.10.0
Jinja2==3.0.1
macos-notifications==0.2.0
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
ntlmlib==0.72
ordereddict==1.1
passlib==1.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pycparser==2.22
Pygments==2.18.0
pyobjc-core==10.2
pyobjc-framework-Cocoa==10.2
pyOpenSSL==22.1.0
requests==2.31.0
rich==13.7.1
service-identity==21.1.0
setuptools==68.0.0
simplejson==3.19.2
six==1.16.0
Twisted==24.3.0
typing_extensions==4.11.0
urllib3==2.0.7
zope.interface==6.4

• Reactor [e.g. select, iocp]

Additional context
Add any other context about the problem here.

[adiroiban]

Thanks for the report.

I don't see how this is related to twisted.

I don't see any twisted code in the traceback.

From what I can see, this is about mac-notifications and that project don't use Twisted.

Can you please share an self-contained example that replicate this error

---

Note that if you are using threads in your Twisted code, those thread should be handled only via Twisted helper API. You should not mix Twited code and threading code without using the Twisted helping API.

---

I am closing this.
If you think this is a bug with twisted please add a complete example that can be used to reproduce this bug and then we can look into this,

Without being able to reproduce this error, it is very hard to fix this.

If you have a fix for this, please submit a pull request and I we can reopen this ticket.

Regards

[michelcrypt4d4mus]

As I said in the bug report all I did was call macos_notifications from opencanary. OpenCanary is a twisted application; here's the .tac file. the bug arises in instantiation just a few lines in when SimpleQueue() constructor is called. i don't have more of the stack trace at hand right now.

Note that if you are using threads in your Twisted code, those thread should be handled only via Twisted helper API. You should not mix Twited code and threading code without using the Twisted helping API.

thanks, i suspect it is related to this, but the code never gets to the thread instantiation. it dies on literally the first line of __init__() in one of the startup classes.

[glyph]

I think there might be more to discuss here.

The fact that this is related to fds_to_keep suggests that Twisted is manipulating some global file descriptor state that it is fighting with Multiprocessing over, and we may have some work to do there.

I am not reopening right away because this really needs a clear reproducer to be actionable enough to be more than a vague "uhhh, maybe, global state, handwave handwave processes, file descriptors" magnet for anxiety or FUD :).

But if someone can do that so we can figure out what, if anything, we are having issues with, that would be great.

[michelcrypt4d4mus]

i think i can create a reproducible example relatively easily; i will try to do so.

[michelcrypt4d4mus]

The fact that this is related to fds_to_keep suggests that Twisted is manipulating some global file descriptor state that it is fighting with Multiprocessing over, and we may have some work to do there.

and fwiw that was my general impression as well.

[michelcrypt4d4mus]

ps i did find a very old issue over in the klein repo where someone seemed to have run into something similar - they were trying to use a package that did its own multiprocessing with a twistd server and getting fds_to_keep issues.

[michelcrypt4d4mus]

Easily reproducible on macOS:

1. Checkout my branch of opencanary: https://github.com/michelcrypt4d4mus/opencanary/tree/twistd_fds_to_keep_reproduce
2. run build_scripts/build_opencanary.sh
3. activate virtual env
4. run twistd -noy bin/opencanary.tac
5. ssh localhost from another window

[michelcrypt4d4mus]

I updated the issue description with the full stack trace instead of just the fragment i had put initially.

[adiroiban]

I am reopening this.

I don't have a macOS VM at hand to try to reproduce this... but I guess that there is enought information here to reproduce it.

It would be awesome to have this error reproduced via GitHub Actions