ERROR running with gunicorn gevent._gevent_cgreenlet.Greenlet.run

[vaibhavpatil123]

• gevent version: 3.0.3
• Python version: 3.10.11
• Operating System: Linux

Description:

Traceback (most recent call last):
File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()
File "/usr/local/lib/python3.10/threading.py", line 1008, in _bootstrap_inner
del _limbo[self]
KeyError: <Thread(OtelBatchSpanProcessor, stopped daemon 139660299675232)>
2024-02-08T13:16:51Z <Greenlet at 0x7f05328d2660: <bound method Thread._bootstrap of <Thread(OtelBatchSpanProcessor, stopped daemon 139660299675232)>>> failed with KeyError

IMPORTANT: Please DO NOT POST SCREENSHOTS OF TEXT. Copy and paste
the actual text. This means: No screenshots of your editor. No
screenshots of tracebacks or console output.

Remember to put tracebacks in literal blocks

What I've run:

REPLACE ME: Paste short, self contained, correct example code (See
http://sscce.org), tracebacks, etc, here

"Put python code in python blocks"

[jamadden]

• That's not a gevent version
• You didn't say what you're doing. A traceback with no context is not a bug report.

[juho-y]

I have this same issue with

• gunicorn 20.10
• gevent 24.2.1
• opentelemetry-sdk 1.23.0
• os: macos sonoma, linux
• does not matter if containerized or not

I have yet to find a consistent repro for this but it goes roughly like this. The Flask app needs to perform a handful of requests, and sometimes, it freezes or throws errors, as mentioned earlier.

This is an issue with the opentelemetry instrumentation with gevent workers.

# gunicorn.config.py


from gevent import monkey

monkey.patch_all()

import multiprocessing
import os


from opentelemetry import metrics, trace  # noqa: E402 monkey patching must be done before importing Instrumentor
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    OTLPSpanExporter,
)
from opentelemetry.sdk.metrics import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    MeterProvider,
)
from opentelemetry.sdk.metrics.export import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    PeriodicExportingMetricReader,
)
from opentelemetry.sdk.resources import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    Resource,
)
from opentelemetry.sdk.trace import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    TracerProvider,
)
from opentelemetry.sdk.trace.export import (  # noqa: E402 monkey patching must be done before importing Instrumentor
    BatchSpanProcessor,
)

workers = 1
worker_class = "gevent"
worker_connections = int(os.getenv("GUNICORN_WORKER_CONNECTIONS", "1000"))
timeout = int(os.getenv("GUNICORN_WORKER_TIMEOUT", "600"))
keepalive = int(os.getenv("GUNICORN_WORKER_KEEPALIVE", "60"))
loglevel = "info"


accesslog = "-"
errorlog = "-"


def post_fork(server, worker):
      server.log.info("Worker spawned (pid: %s)", worker.pid)
      resource = Resource.create(attributes={"service.name": os.getenv("OTEL_SERVICE_NAME", "lih")})
      trace.set_tracer_provider(TracerProvider(resource=resource))
      span_processor = BatchSpanProcessor(OTLPSpanExporter(insecure=True))
      trace.get_tracer_provider().add_span_processor(span_processor)

      reader = PeriodicExportingMetricReader(OTLPMetricExporter())
      metrics.set_meter_provider(
          MeterProvider(
              resource=resource,
              metric_readers=[reader],
          )
      )

# app.py
from flask import Flask, request
from opentelemetry.instrumentation.flask import FlaskInstrumentor
import os


app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)


@app.route("/", methods=["POST"])
def process_request():
    data = request.get_json()

    do_something_with_data(data)



if __name__ == "__main__":
    app.run(debug=os.getenv("FLASK_DEBUG", "False").lower() == "true")

FROM python:3.10

ENV PYTHONBUFFERED 1

COPY . .

RUN pip install -r requirements.txt

EXPOSE 5000

CMD ["gunicorn", "-c", "gunicorn.config.py", "--bind", "0.0.0.0:5000", "app:app"]