Call StepFunction SendTaskSuccess once then got: Task Timed Out: 'Provided task does not exist anymore', but succeeded eventually

[NianLi71]

Describe the bug

I have code that consumes messages from SQS stand queue, each message will make a call to StepFunction SendTaskSuccess. I checked log that one message only called StepFunction SendTaskSuccess once with valid task token and got:

Failed to process with exception: An error occurred (TaskTimedOut) when calling the SendTaskSuccess operation: Task Timed Out: 'Provided task does not exist anymore'

I also saved the message id in DynamoDB, it was the same message that updated the DB item which caused TaskTimedOut
Looks like boto3 had made the first attempt to send StepFunction task token and the token expired, but eventually the SendTaskSuccess operation succeeded even with exception like above, and StepFunction successfully received the task token.

Any boto3 inside retry mechanism leads to this issue?

Expected Behavior

Should be no exception:

Failed to process with exception: An error occurred (TaskTimedOut) when calling the SendTaskSuccess operation: Task Timed Out: 'Provided task does not exist anymore'

when making first call

Current Behavior

Got exception

Failed to process with exception: An error occurred (TaskTimedOut) when calling the SendTaskSuccess operation: Task Timed Out: 'Provided task does not exist anymore'

even with first call to SendTaskSuccess, but eventually the SendTaskSuccess operation succeeded.

Reproduction Steps

The error was random, I tried to send about greater than 10K requests then there was one TaskTimedOut exception.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

Boto3==1.34.50, BotoCore==1.34.50

Environment details (OS name and version, etc.)

AWS Lambda Python 3.9 x86_64

[tim-finnigan]

Hello - thanks for reaching out and for your patience here. The send_task_status command involves a call to the underlying SendTaskStatus API. Therefore if there's an issue with the behavior here then it's likely something we'd need to escalate to the Step Functions team.

The TaskTimedOut exception indicates:

The task token has either expired or the task associated with the token has already been closed.

As a service exception this would be caught with the ClientError Botocore exception. The retry behavior depends on however you have configured retries.

If the error was random and only 1 in 10k as you mentioned, then this may have just been caused by something transient like a network issue. Maybe get_execution_history would help provide more context. For us to investigate this further I think we need a code snippet to reproduce the issue, and debug logs (using boto3.set_stream_logger('') to get more insight into what's going on.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.