Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idempotent Workflow Bug: Duplicate IVS Channels Created by IVSCreateChannelsLambdaFunction #424

Open
NullPointer4096 opened this issue Apr 14, 2023 · 0 comments

Comments

@NullPointer4096
Copy link

Hello,

The ivs-create-channels.lambda_handler function is asynchronously invoked by CloudFormation to create, update, or delete IVS channels and maintain the video-to-channel mappings in SSM.

For each video_file_key, the workflow is that if there’s no IVS channel associated with it in SSM, create a channel with a name generated from the current time and record the mapping in SSM. If an error occurs after the channel is created but before the mapping is updated (L156 or L164), the function retries and creates a new channel with a different name in the second execution. This creates duplicate IVS channels, leaving the old channel unused and undeleted. Though unlikely to happen, the triggering error includes any software, hardware, and network errors that would cause the asynchronously invoked function to retry. The expected behavior is that the function should not create duplicate IVS channels at retries after a partial execution (i.e., created channels in the first run but did not update the mapping stored in SSM).

Suggested fix:
First, make the channel_name deterministic across retries. Maybe replace the ever-changing timestamp with the deterministic aws_request_id passed along in the context object. Second, before creating a channel, query whether a channel of this name has been created through IVS’ list_channels API. If it has been created, record the association in SSM. Otherwise, create the channel and record it to SSM. This approach ensures that only one channel is created and associated with the video, preventing the creation of unused channels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant