[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations #771

valebedu · 2024-04-19T10:17:08Z

Describe the bug

When deploying an API using type-safe-api with a really large integrations object the PrepareSpecCustomResource fail with 413 because of large payload in input

Error:

1115107 byte payload is too large for the Event invocation type (limit 262144 bytes) (Service: AWSLambda; Status Code: 413; Error Code: RequestEntityTooLargeException; Request ID: 9f16fddd-3500-4106-aba5-07d6398ba67d; Proxy: null)

Expected Behavior

The PrepareSpecCustomResource process integrations successfully and deploy continue

Current Behavior

The PrepareSpecCustomResource fail and stop the deploy

Reproduction Steps

In order to reproduce the error it's pretty simple you need to make a really large integrations object and deploy this api

Here is the way I'm facing this issue in a real production scenario:

Follow the documentation in order to create the base type-safe-api project in TS
In the construct myapi.ts add the following constant and functions in order to create huge integration template responses

const RESPONSE_TEMPLATE = `#set($code = $input.body.split('<Code>')[1].split('</Code>')[0])
#set($message = $input.body.split('<Message>')[1].split('</Message>')[0])

#if($code == 'AccessControlListNotSupported')#set($errorCode = 'STOR1')
#elseif($code == 'AccessDenied')#set($errorCode = 'STOR2')
#elseif($code == 'AccessPointAlreadyOwnedByYou')#set($errorCode = 'STOR3')
#elseif($code == 'AccountProblem')#set($errorCode = 'STOR4')
#elseif($code == 'AllAccessDisabled')#set($errorCode = 'STOR5')
#elseif($code == 'AmbiguousGrantByEmailAddress')#set($errorCode = 'STOR6')
#elseif($code == 'AuthorizationHeaderMalformed')#set($errorCode = 'STOR7')
#elseif($code == 'BadDigest')#set($errorCode = 'STOR8')
#elseif($code == 'BucketAlreadyExists')#set($errorCode = 'STOR9')
#elseif($code == 'BucketAlreadyOwnedByYou')#set($errorCode = 'STOR10')
#elseif($code == 'BucketNotEmpty')#set($errorCode = 'STOR11')
#elseif($code == 'ClientTokenConflict')#set($errorCode = 'STOR12')
#elseif($code == 'CredentialsNotSupported')#set($errorCode = 'STOR13')
#elseif($code == 'CrossLocationLoggingProhibited')#set($errorCode = 'STOR14')
#elseif($code == 'EntityTooSmall')#set($errorCode = 'STOR15')
#elseif($code == 'EntityTooLarge')#set($errorCode = 'STOR16')
#elseif($code == 'ExpiredToken')#set($errorCode = 'STOR17')
#elseif($code == 'IllegalLocationConstraintException')#set($errorCode = 'STOR18')
#elseif($code == 'IllegalVersioningConfigurationException')#set($errorCode = 'STOR19')
#elseif($code == 'IncompleteBody')#set($errorCode = 'STOR20')
#elseif($code == 'IncorrectNumberOfFilesInPostRequest')#set($errorCode = 'STOR21')
#elseif($code == 'InlineDataTooLarge')#set($errorCode = 'STOR22')
#elseif($code == 'InternalError')#set($errorCode = 'STOR23')
#elseif($code == 'InvalidAccessKeyId')#set($errorCode = 'STOR24')
#elseif($code == 'InvalidAccessPoint')#set($errorCode = 'STOR25')
#elseif($code == 'InvalidAccessPointAliasError')#set($errorCode = 'STOR26')
#elseif($code == 'InvalidAddressingHeader')#set($errorCode = 'STOR27')
#elseif($code == 'InvalidArgument')#set($errorCode = 'STOR28')
#elseif($code == 'InvalidBucketAclWithObjectOwnership')#set($errorCode = 'STOR29')
#elseif($code == 'InvalidBucketName')#set($errorCode = 'STOR30')
#elseif($code == 'InvalidBucketState')#set($errorCode = 'STOR31')
#elseif($code == 'InvalidDigest')#set($errorCode = 'STOR32')
#elseif($code == 'InvalidEncryptionAlgorithmError')#set($errorCode = 'STOR33')
#elseif($code == 'InvalidLocationConstraint')#set($errorCode = 'STOR34')
#elseif($code == 'InvalidObjectState')#set($errorCode = 'STOR35')
#elseif($code == 'InvalidPart')#set($errorCode = 'STOR36')
#elseif($code == 'InvalidPartOrder')#set($errorCode = 'STOR37')
#elseif($code == 'InvalidPayer')#set($errorCode = 'STOR38')
#elseif($code == 'InvalidPolicyDocument')#set($errorCode = 'STOR39')
#elseif($code == 'InvalidRange')#set($errorCode = 'STOR40')
#elseif($code == 'InvalidRequest')#set($errorCode = 'STOR41')
#elseif($code == 'InvalidSecurity')#set($errorCode = 'STOR42')
#elseif($code == 'InvalidSOAPRequest')#set($errorCode = 'STOR43')
#elseif($code == 'InvalidStorageClass')#set($errorCode = 'STOR44')
#elseif($code == 'InvalidTargetBucketForLogging')#set($errorCode = 'STOR45')
#elseif($code == 'InvalidToken')#set($errorCode = 'STOR46')
#elseif($code == 'InvalidURI')#set($errorCode = 'STOR47')
#elseif($code == 'KeyTooLongError')#set($errorCode = 'STOR48')
#elseif($code == 'MalformedACLError')#set($errorCode = 'STOR49')
#elseif($code == 'MalformedPOSTRequest')#set($errorCode = 'STOR50')
#elseif($code == 'MalformedXML')#set($errorCode = 'STOR51')
#elseif($code == 'MaxMessageLengthExceeded')#set($errorCode = 'STOR52')
#elseif($code == 'MaxPostPreDataLengthExceededError')#set($errorCode = 'STOR53')
#elseif($code == 'MetadataTooLarge')#set($errorCode = 'STOR54')
#elseif($code == 'MethodNotAllowed')#set($errorCode = 'STOR55')
#elseif($code == 'MissingAttachment')#set($errorCode = 'STOR56')
#elseif($code == 'MissingContentLength')#set($errorCode = 'STOR57')
#elseif($code == 'MissingRequestBodyError')#set($errorCode = 'STOR58')
#elseif($code == 'MissingSecurityElement')#set($errorCode = 'STOR59')
#elseif($code == 'MissingSecurityHeader')#set($errorCode = 'STOR60')
#elseif($code == 'NoLoggingStatusForKey')#set($errorCode = 'STOR61')
#elseif($code == 'NoSuchBucket')#set($errorCode = 'STOR62')
#elseif($code == 'NoSuchBucketPolicy')#set($errorCode = 'STOR63')
#elseif($code == 'NoSuchCORSConfiguration')#set($errorCode = 'STOR64')
#elseif($code == 'NoSuchKey')#set($errorCode = 'STOR65')
#elseif($code == 'NoSuchLifecycleConfiguration')#set($errorCode = 'STOR66')
#elseif($code == 'NoSuchMultiRegionAccessPoint')#set($errorCode = 'STOR67')
#elseif($code == 'NoSuchWebsiteConfiguration')#set($errorCode = 'STOR68')
#elseif($code == 'NoSuchTagSet')#set($errorCode = 'STOR69')
#elseif($code == 'NoSuchUpload')#set($errorCode = 'STOR70')
#elseif($code == 'NoSuchVersion')#set($errorCode = 'STOR71')
#elseif($code == 'NotImplemented')#set($errorCode = 'STOR72')
#elseif($code == 'NotModified')#set($errorCode = 'STOR73')
#elseif($code == 'NotSignedUp')#set($errorCode = 'STOR74')
#elseif($code == 'OwnershipControlsNotFoundError')#set($errorCode = 'STOR75')
#elseif($code == 'OperationAborted')#set($errorCode = 'STOR76')
#elseif($code == 'PermanentRedirect')#set($errorCode = 'STOR77')
#elseif($code == 'PreconditionFailed')#set($errorCode = 'STOR78')
#elseif($code == 'Redirect')#set($errorCode = 'STOR79')
#elseif($code == 'RequestHeaderSectionTooLarge')#set($errorCode = 'STOR80')
#elseif($code == 'RequestIsNotMultiPartContent')#set($errorCode = 'STOR81')
#elseif($code == 'RequestTimeout')#set($errorCode = 'STOR82')
#elseif($code == 'RequestTimeTooSkewed')#set($errorCode = 'STOR83')
#elseif($code == 'RequestTorrentOfBucketError')#set($errorCode = 'STOR84')
#elseif($code == 'RestoreAlreadyInProgress')#set($errorCode = 'STOR85')
#elseif($code == 'ServerSideEncryptionConfigurationNotFoundError')#set($errorCode = 'STOR86')
#elseif($code == 'ServiceUnavailable')#set($errorCode = 'STOR87')
#elseif($code == 'SignatureDoesNotMatch')#set($errorCode = 'STOR88')
#elseif($code == 'SlowDown')#set($errorCode = 'STOR89')
#elseif($code == '503 SlowDown')#set($errorCode = 'STOR90')
#elseif($code == 'TemporaryRedirect')#set($errorCode = 'STOR91')
#elseif($code == 'TokenRefreshRequired')#set($errorCode = 'STOR92')
#elseif($code == 'TooManyAccessPoints')#set($errorCode = 'STOR93')
#elseif($code == 'TooManyBuckets')#set($errorCode = 'STOR94')
#elseif($code == 'TooManyMultiRegionAccessPointregionsError')#set($errorCode = 'STOR95')
#elseif($code == 'TooManyMultiRegionAccessPoints')#set($errorCode = 'STOR96')
#elseif($code == 'UnexpectedContent')#set($errorCode = 'STOR97')
#elseif($code == 'UnresolvableGrantByEmailAddress')#set($errorCode = 'STOR98')
#elseif($code == 'UserKeyMustBeSpecified')#set($errorCode = 'STOR99')
#elseif($code == 'NoSuchAccessPoint')#set($errorCode = 'STOR100')
#elseif($code == 'InvalidTag')#set($errorCode = 'STOR101')
#elseif($code == 'MalformedPolicy')#set($errorCode = 'STOR102')
#{else}#set($errorCode = 'STORX')#end

{"error": {"type": "storage_error","code": "$errorCode", "message": "$util.escapeJavaScript($message).replaceAll("\\'","'")"}, "meta": {}}`

function errorResponseSet(
  responseTemplate: string,
): typesafeapi.IntegrationResponseSet {
  return typesafeapi.IntegrationResponseSets.custom({
    responses: {
      ...Object.fromEntries(
        Array.of('400', '403', '404', '500').map((statusCode) => [
          statusCode,
          {
            statusCode: statusCode,
            responseParameters: {},
            responseTemplates: {
              'application/json': responseTemplate,
            },
          },
        ]),
      ),
      '(4|5)\\d{2}': {
        statusCode: '500',
        responseParameters: {},
        responseTemplates: {
          'application/json': responseTemplate,
        },
      },
    },
  })
}

function cacheResponseSet(
  responseTemplate: string,
): typesafeapi.IntegrationResponseSet {
  return typesafeapi.IntegrationResponseSets.composite(
    typesafeapi.IntegrationResponseSets.defaultPassthrough(),
    errorResponseSet(responseTemplate),
  )
}

Then in constructor create a CfnMapping in order to keep the Cloudformation Template as small as possible (this will allow to have a virtually big integrations object while keeping the cloudformation template low because it will use FindInMap)

const defaultMapping = new CfnMapping(scope, 'DefaultMapping', {
  mapping: {
    default: {
      responseTemplate: RESPONSE_TEMPLATE,
    },
  },
})

Then in super.integrations create a lot of integration with similar props (you'll need to create in the OpenAPI spec a lot of fake route in order to do that)

getA: {
  integration: typesafeapi.Integrations.s3({
    bucket: props.cacheBucket,
    method: 'get',
    path: 'a.json',
    integrationResponseSet: cacheResponseSet(
      defaultMapping.findInMap('default', 'responseTemplate'),
    ),
  }),
  authorizer: typesafeapi.Authorizers.none(),
},
...
getZ: {
  integration: typesafeapi.Integrations.s3({
    bucket: props.cacheBucket,
    method: 'get',
    path: 'z.json',
    integrationResponseSet: cacheResponseSet(
      defaultMapping.findInMap('default', 'responseTemplate'),
    ),
  }),
  authorizer: typesafeapi.Authorizers.none(),
},

Possible Solution

In the PrepareSpecCustomResource there are several properties:

{
  inputSpecLocation: {
    bucket: inputSpecAsset.bucket.bucketName,
    key: inputSpecAsset.s3ObjectKey,
  },
  outputSpecLocation: {
    bucket: inputSpecAsset.bucket.bucketName,
    key: preparedSpecOutputKeyPrefix,
  },
  ...prepareSpecOptions,
};

and in prepareSpecOptions integrations and everything else about the api is passed directly, so in a context of a large integrations object, the process fail everytime.

A workaround could be to write all of the info into a bucket and then read the bucket.

Additional Information/Context

No response

PDK version used

v0.23.37

What languages are you seeing this issue on?

Typescript

Environment details (OS name and version, etc.)

macOS Sonoma 14.4.1 Apple M1 Pro

The text was updated successfully, but these errors were encountered:

agdimech · 2024-04-24T02:59:11Z

Hi @valebedu - thanks for raising this. I agree with your proposal - would you have capacity to raise a backwards compat PR to resolve this or would you be looking for us to add this for you? The only reason I bring this up is that we won't be able to realistically add this into the AWS PDK for another couple of weeks.

valebedu added bug Something isn't working needs-triage labels Apr 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations #771

[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations #771

valebedu commented Apr 19, 2024 •

edited

agdimech commented Apr 24, 2024 •

edited

[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations #771

[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations #771

Comments

valebedu commented Apr 19, 2024 • edited

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

PDK version used

What languages are you seeing this issue on?

Environment details (OS name and version, etc.)

agdimech commented Apr 24, 2024 • edited

valebedu commented Apr 19, 2024 •

edited

agdimech commented Apr 24, 2024 •

edited