caproto IOCs on Windows Python 3.12 have issues

[klauer]

Windows Python 3.12 failure is legitimate and locally reproducible (as well as on CI).

Copied from my comment in #829:

• The accuracy of the claim below is not guaranteed - it was based on some debugging efforts which I ultimately gave up on

How it's failing is a little bizarre.

caproto/tests/test_threading_client.py::test_unsubscribe_all[caproto]

This test:

• Makes 2 subscriptions of different data types to the same PV.
• Writes a few values
• Double-checks that we got the number of subscriptions expected

On Windows Python 3.12, the server only sends back the first data type that was requested. This is not the case on other platforms.

I'm at a loss as to how it's happening (TODO look into this again at some point):

• The two subscription events (EventAddResponse) are put in as weakrefs into to the queue (circuit.subscription_queue.put)
• The weakrefs are not destroyed (checked via logs; they have a hard reference in "unexpired updates")
• Only one subscription makes it out of the queue (ref = await self.get_from_sub_queue(...) -> circuit.subscription_queue.get() wrapped with ensure_future to add timeout support)

Log

DEBUG    caproto.circ:_circuit.py:177 EventAddRequest(data_type=<ChannelType.INT: 1>, data_count=0, sid=0, subscriptionid=0, low=0.0, high=0.0, to=0.0, mask=13)
DEBUG    caproto.circ:_circuit.py:177 EventAddRequest(data_type=<ChannelType.STRING: 0>, data_count=0, sid=0, subscriptionid=1, low=0.0, high=0.0, to=0.0, mask=13)

Initial sub value for INT only comes back:

DEBUG    caproto.ch:client.py:2185 'a17b46ee:int': EventAddResponse(data=array([96], dtype='>i2'), data_type=<ChannelType.INT: 1>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), subscriptionid=0, metadata=None)
DEBUG    caproto.circ:client.py:1484 EventAddResponse(data=array([96], dtype='>i2'), data_type=<ChannelType.INT: 1>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), subscriptionid=0, metadata=None)

Write 123:

DEBUG    caproto.circ:_circuit.py:177 WriteNotifyRequest(data=array([123], dtype='>i4'), data_type=<ChannelType.LONG: 5>, data_count=1, sid=0, ioid=0, metadata=None)
DEBUG    caproto.circ:client.py:1484 WriteNotifyResponse(data_type=<ChannelType.LONG: 5>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), ioid=0)

See INT monitor event but not string one:

DEBUG    caproto.circ:_circuit.py:177 WriteNotifyRequest(data=array([456], dtype='>i4'), data_type=<ChannelType.LONG: 5>, data_count=1, sid=0, ioid=1, metadata=None)
DEBUG    caproto.ch:client.py:2185 'a17b46ee:int': EventAddResponse(data=array([123], dtype='>i2'), data_type=<ChannelType.INT: 1>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), subscriptionid=0, metadata=None)
DEBUG    caproto.circ:client.py:1484 EventAddResponse(data=array([123], dtype='>i2'), data_type=<ChannelType.INT: 1>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), subscriptionid=0, metadata=None)
DEBUG    caproto.circ:client.py:1484 WriteNotifyResponse(data_type=<ChannelType.LONG: 5>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), ioid=1)


DEBUG    caproto.ch:client.py:2185 'a17b46ee:int': EventAddResponse(data=array([456], dtype='>i2'), data_type=<ChannelType.INT: 1>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), subscriptionid=0, metadata=None)
DEBUG    caproto.circ:client.py:1484 EventAddResponse(data=array([456], dtype='>i2'), data_type=<ChannelType.INT: 1>, data_count=1, status=CAStatusCode(name='ECA_NORMAL', code=0, code_with_severity=1, severity=<CASeverity.SUCCESS: 1>, success=1, defunct=False, description='Normal successful completion'), subscriptionid=0, metadata=None)