Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nats-py broken while running opentelemetry instrumentation 0.44b0 #549

Open
bvvstab opened this issue Mar 19, 2024 · 0 comments
Open

nats-py broken while running opentelemetry instrumentation 0.44b0 #549

bvvstab opened this issue Mar 19, 2024 · 0 comments
Labels
defect Suspected defect such as a bug or regression

Comments

@bvvstab
Copy link

bvvstab commented Mar 19, 2024

Observed behavior

We are running nats consumers in django commands, on a kubernetes cluster with the opentelemetry operator set up.

The consumers are looking (simplified) like this:

    sub = await conn.pull_subscribe(
         ...,
        config=ConsumerConfig(
            inactive_threshold=60 * 60 * 24,
        ),
    )
    while True:
        try:
            msgs = await sub.fetch(batch=10, timeout=10)
            for msg in msgs:
                await handle_event(msg, handler)
                await msg.ack()
        except TimeoutError:
            await asyncio.sleep(10)

We use opentelemetry-operators automatic injections by adding the annotation:

annotations:
   instrumentation.opentelemetry.io/inject-python: "observability/instrumentation"

When running otel instrumentation 0.42b0 (I think), there has been no interruption, but when upgrading to 0.44b0, the consumers all fail with:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/concurrent/futures/process.py", line 263, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/lib/bvvs/events/management/commands/run_event_worker.py", line 10, in run_async_worker
    asyncio.run(run_worker())
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/src/lib/bvvs/events/worker.py", line 75, in run_worker
    await pull_and_handle(conn)
  File "/src/lib/bvvs/events/worker.py", line 51, in pull_and_handle
    msgs = await sub.fetch(batch=10, timeout=10)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 939, in fetch
    msgs = await self._fetch_n(batch, expires, timeout, heartbeat)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 1093, in _fetch_n
    msg = await self._sub.next_msg(timeout=deadline)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/nats/aio/subscription.py", line 199, in next_msg
    self._pending_size -= len(msg.data)
                              ^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'data'

Expected behavior

Updating opentelemetry should not affect nats.

Server and client version

server unknown (managed at Scaleway)
nats-py 2.7.2

Host environment

No response

Steps to reproduce

No response

@bvvstab bvvstab added the defect Suspected defect such as a bug or regression label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Development

No branches or pull requests

1 participant