Enhance OTEL testing to capture and verify Cancellation Requests and Non-Decoupled model inference.

[indrajit96]

Added tests for

1. For request cancellations, make sure spans are ended properly in various scenarios
2. Verify tracing behavior, after this PR: Allow non-decoupled model to send response and FINAL flag separately #6017

Tests added

1. test_non_decoupled() : Verify tracing for a non-decoupled inference
2. test_grpc_trace_all_input_required_model_cancel() : Verify trace after an inference request is cancelled in COMPUTE Phase
3. test_grpc_trace_model_cancel_in_queue() : Verify trace after an inference request is cancelled in QUEUE before COMPUTE Phase

[oandreeva-nv]

Could you please add a description, clarifying what tests were added?

[oandreeva-nv]

Could you please attach a picture of a trace as displayed in jaeger with cancelled request.

Another question, have you considered cases when request was cancelled, when it was in a queue and when it was already in a compute stage?

[indrajit96]

Could you please attach a picture of a trace as displayed in jaeger with cancelled request.

Another question, have you considered cases when request was cancelled, when it was in a queue and when it was already in a compute stage?

Could you please attach a picture of a trace as displayed in jaeger with cancelled request.

Another question, have you considered cases when request was cancelled, when it was in a queue and when it was already in a compute stage?

Fixed this, added a model which waits for a delay before executing hence cancellation request is recieved before the execution starts

[rmccorm4]

Can you update the PR title to be more descriptive? (cancellation, decoupled responses, etc. rather than JIRA ticket number)

[oandreeva-nv]

Could you please clarify what delay in the model? and if it is a model related, why are we using COLLECTOR_TIMEOUT variable?

[indrajit96]

We want to capture a cancellation request traces WHILE the inference is in the COMPUTE stage.
Because the model "input_all_required" has a delay/wait in the compute phase so the cancellation request can be send while the request is waiting in the compute phase.
The idea here is to wait before we try and read the traces from the file.

If we do not wait, the file in empty without traces.
Updated the comments in the test to reflect his better.

[oandreeva-nv]

I think it's in a good state. Let's address Ryan's comment and provide a clear description for 3 tests, added in this PR, to the PR description

[oandreeva-nv]

LGTM! Thank you for this hard work!