Pytest fails consistently in .devcontainer/post-start.sh

[agardnerIT]

The issue has been identified and a fix in in progress, this is just to track the investigation of hte issue and how the root cause was identified + next steps.

Describe the bug

The .devcontainer/post-start.sh script fails consistently due to pytest failing.

Findings

Pytest fails because one of the tests fail: test_wait_for_collector. To be more specific, the test case doesn't just fail, it throws exceptions which cause a crash.

The Problem

The issue is that in order to streamline the codespaces setup, port 4318 (the HTTP endpoint of the OTEL collector) was removed, leaving only 4317 (gRPC).

test_wait_for_collector was therefore using the requests module to GET an endpoint: http://localhost:4317/v1/logs. The response from this endpoint (being a gRPC endpoint) made no sense to requests (which is essentially a wrapper around cURL and thus expecting an HTTP-style response).

A Possible solution

My first thought was to look for a gRPC health check endpoint on the OTEL collector and use that instead. The gRPC spec does recommend an endpoint.

But, it is not yet available. There is a PR for it.

The Solution

There are a few potential solutions, from "quick and hacky" to "best solution".

The band-aid solution is to re-open 4318 and have pytest wait for that endpoint & push traces there.

The second stage would be to move every service over to only 4318 and thus we can go back to only one port being open (ie. 4318).

Since 4317 and/ or 4318 only need to be open for pytest (which runs on the container and not inside the Kind cluster, the long term best solution is probably to find a way to trigger pytest from inside the cluster. Thus neither 4317 or 4318 need to be exposed at all.