As of 11:00 EST on 5/3, we have been seeing a similar error with Python 3.10, CDK 2.134.0, using an AwsSdkCall for SSM's getParameter action. In our case the error is Package @aws-sdk/client-ssm does not exist.

cr.AwsCustomResource(
    self,
    "get_parameter",
    on_update=cr.AwsSdkCall(
        service="SSM",
        action="getParameter",
        parameters={
            "Name": parameter_name,
            "WithDecryption": True,
        },
        physical_resource_id=cr.PhysicalResourceId.of(
            str(datetime.utcnow()),
        ),
        region=region,
    ),
    policy=cr.AwsCustomResourcePolicy.from_sdk_calls(
        resources=[
            Stack.of(self).format_arn(
                service="ssm",
                region=region,
                resource="parameter",
                resource_name=parameter_name.lstrip("/"),
            )
        ]
    ),
)

The issue also appears to be intermittent for us.

For now, un-setting install_latest_aws_sdk seems to have stabilized our configuration (based on ~3 repeated deployments)... But I feel like it might be an intermittency thing / luck-of-the-draw, rather than a real remedy. Our full source code & patch commit available here

@glitchassassin it looks like you're not using the install_latest_aws_sdk option though right? And still seeing the issue?

Correct, we are not.

On Friday, it failed on 2/6 deploys. Today we've had four successful releases so far and no failures. I'm configuring logging on the AwsSdkCall in hopes of capturing more details if it happens again

Aha, tracked down some logs from Friday! They showed up by default in a Cloudwatch log group named /aws/lambda/[stack_name]-AWS[random hexadecimal]

Installing latest AWS SDK v3: @aws-sdk/client-ssm
Failed to install latest AWS SDK v3. Falling back to pre-installed version. Error: SyntaxError: Error parsing /tmp/node_modules/@smithy/shared-ini-file-loader/package.json: Unexpected end of JSON input

In another instance:

Installing latest AWS SDK v3: @aws-sdk/client-ssm
Failed to install latest AWS SDK v3. Falling back to pre-installed version. Error: Error: Cannot find module '@smithy/shared-ini-file-loader'
Require stack:

• /tmp/node_modules/@smithy/node-config-provider/dist-cjs/index.js
• /tmp/node_modules/@smithy/middleware-endpoint/dist-cjs/adaptors/getEndpointFromConfig.js
• /tmp/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js
• /tmp/node_modules/@smithy/core/dist-cjs/index.js
• /tmp/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js
• /var/task/index.js
• /var/runtime/index.mjs

It seems like each time this runs, there's an initial attempt to install the SDK which always times out after 120 seconds (based on ResourceProperties in the logs, InstallLatestAwsSdk is true even though it isn't explicitly set in our code). The lambda is immediately invoked again, and this time the install either succeeds or fails in under a minute. If it fails, it says it is falling back to pre-installed version.

After the install, an Update request is logged, and it returns the parameter it's supposed to be fetching correctly (whether the install failed or succeeded).

Then, in some cases, there is a second Update request in the logs a couple minutes later, and that is where the "Package does not exist" error gets thrown. The request is identical to the first Update request except that the physicalResourceId is different (it's using the current date/time as described here.)

After reviewing our deployment logs, this seems to only have happened when we had back-to-back deployments within a couple minutes of each other, so the second deployment's Update request hits the same running lambda instance that was created by the first deployment.

It looks like when the Lambda doesn't get cleaned up after an install failure, the next Update request fails.

Based on this:

I wonder if the initial npm install failure is leaving /tmp/node_modules in an invalid state, but a subsequent npm install fails to detect the issue and thinks everything is installed?

Nope! It's actually failing on the require, not on the npm install command. So at this point installedSdk[packageName] is true. Next time it runs, the handler skips trying to install and falls through to the next block on the if statement:

But there's no try/catch here, so this time when the require fails, it doesn't fall back to the pre-installed version.

Drafting a PR with a fix

thanks @athewsey for reporting this issue. There have been multiple incidences of this issue reported by the customers recently

Thanks @glitchassassin for submitting a PR.

Hi, any update on this?
we have started getting this as well when deleting Events Rule for some reason
"Package @aws-sdk/client-cloudwatch-logs does not exist"

This is holding our pipelines right now from fully passing

Waiting on some guidance on the failing integration tests on the PR - I'm not sure how to resolve the build issues