LambdaAsyncClient leads to OutOfMemoryErrors under a high load

[maxhanel]

Describe the bug

The LambdaAsyncClient consumes a lot of JVM Memory and leads to an OutOfMemory Error.
Anybody a Idea, why this happens?

Expected Behavior

The LambdaAsyncClient should consume less JVM Memory.

Current Behavior

When we make requests with the Lambda Async Client, the JVM heap immediately jumps to more than 90 percent and stops at this level or the application terminates:
Terminating due to java.lang.OutOfMemoryError: Java heap space
We are using this configuration for the LambdaAsyncClient:

    @Bean
    fun createLambdaAsyncClient(): LambdaAsyncClient {
        val httpClient = NettyNioAsyncHttpClient.builder()
            .maxConcurrency(100)
            .maxPendingConnectionAcquires(100)
            .connectionTimeout(Duration.ofSeconds(10))
            .connectionMaxIdleTime(Duration.ofSeconds(5))
            .readTimeout(Duration.ofSeconds(301))
            .writeTimeout(Duration.ofSeconds(301))
            .build()

        val clientCfg: ClientOverrideConfiguration = ClientOverrideConfiguration.builder()
            .retryPolicy(RetryPolicy.defaultRetryPolicy())
            .addMetricPublisher(AwsSdkMetricPublisher(meterRegistry))
            .build()

        return LambdaAsyncClient.builder()
            .region(Region.EU_CENTRAL_1)
            .httpClient(httpClient)
            .overrideConfiguration(clientCfg)
            .build()
    }

The Invoke Request looks like this:

val response = lambdaClient.invoke(
    InvokeRequest.builder()
        .functionName(functionName)
        .payload(SdkBytes.fromByteArray(bytes))
        .invocationType(
            InvocationType.REQUEST_RESPONSE,
        ).build(),
    ).await()

Afer executing about 200 Requests the heap increases expcetionally, after that decreases, but increases in a slow way.

Reproduction Steps

I was not able to reproduce this locally.
Our application processes a queue with 100 workers. A lambda request is made at least once with every job that is processed by the queue.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.25.34

JDK version used

jdk-17.0.9+9

Operating System and version

Linux/ARM64