IndexOutOfBoundsException in LZ4Codec

[nigelmenger]

Expected behavior
Redisson does not throw IndexOutOfBoundsException.

Actual behavior
Redisson sometimes throws IndexOutOfBoundsException:

{ 
   exception: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(1): UnpooledSlicedByteBuf(ridx: 0, widx: 1, cap: 1/1, unwrapped: PooledUnsafeDirectByteBuf(ridx: 7, widx: 7, cap: 36))
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
	at io.netty.buffer.AbstractByteBuf.readInt(AbstractByteBuf.java:809)
	at org.redisson.codec.LZ4Codec$1.decode(LZ4Codec.java:72)
	at o.r.c.handler.CommandDecoder.decode(CommandDecoder.java:436)
	at o.r.c.handler.CommandDecoder.decodeCommand(CommandDecoder.java:216)
	at o.r.c.handler.CommandDecoder.decode(CommandDecoder.java:144)
	at o.r.c.handler.CommandDecoder.decode(CommandDecoder.java:120)
	at i.n.h.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at i.n.h.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
	at i.n.h.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
   level: ERROR
   logger: org.redisson.client.handler.CommandDecoder
   message: Unable to decode data. channel: [id: 0x5ef4f8d2, L:/10.0.6.222:46646 - R:10.2.195.10/10.2.195.10:6379], reply: ReplayingDecoderByteBuf(ridx=7, widx=7), command: (GET), promise: java.util.concurrent.CompletableFuture@18a00660[Not completed, 1 dependents], params: [stuck-job-count]
   thread: redisson-netty-3-27
   timestamp: 1714541415762
}

Followed by:

{ 
   exception: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(1): UnpooledSlicedByteBuf(ridx: 0, widx: 1, cap: 1/1, unwrapped: PooledUnsafeDirectByteBuf(ridx: 7, widx: 7, cap: 36))
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
	at io.netty.buffer.AbstractByteBuf.readInt(AbstractByteBuf.java:809)
	at org.redisson.codec.LZ4Codec$1.decode(LZ4Codec.java:72)
	at o.r.c.handler.CommandDecoder.decode(CommandDecoder.java:436)
	at o.r.c.handler.CommandDecoder.decodeCommand(CommandDecoder.java:216)
	at o.r.c.handler.CommandDecoder.decode(CommandDecoder.java:144)
	at o.r.c.handler.CommandDecoder.decode(CommandDecoder.java:120)
	at i.n.h.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at i.n.h.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
	at i.n.h.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
Wrapped by: io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(1): UnpooledSlicedByteBuf(ridx: 0, widx: 1, cap: 1/1, unwrapped: PooledUnsafeDirectByteBuf(ridx: 7, widx: 7, cap: 36))
	at i.n.h.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:421)
	at i.n.h.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at i.n.c.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at i.n.c.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
	at i.n.h.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at i.n.h.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
   level: ERROR
   logger: org.redisson.client.handler.ErrorsLoggingHandler
   message: Exception occured. Channel: [id: 0x5ef4f8d2, L:/10.0.6.222:46646 - R:10.2.195.10/10.2.195.10:6379]
   thread: redisson-netty-3-27
   timestamp: 1714541415774
}

Steps to reproduce or test case
This happened on deployment of new pods which contain Redisson. No changes to how the object was serialized between versions that were deployed. Object is a:

RBucket<Int?> = redissonClient.getBucket("stuck-job-count")

Redis version
ElastiCache engine version 7.1.0

Redisson version
3.27.2

Redisson configuration

Redisson.create(
        Config().apply {
            val baseConfig = useClusterServers().apply {
                    password = redisPassword
                    nodeAddresses = mutableListOf(redisUrl)
                    this.retryAttempts = 100
                    val redissonTimerRounding = Duration.ofMillis(100)
                    val max100MsJitters = retryMaxJitter.dividedBy(redissonTimerRounding)
                    val retryJitter = redissonTimerRounding.multipliedBy((-max100MsJitters..max100MsJitters).random())
                    // Adjust retry attempts and interval with jitter to avoid RedisNodeNotFoundException on failover
                    // See https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
                    // When all pods use the same retry interval, the Redis nodes get overwhelmed with requests
                    // Exponential backoff is not available in Redisson
                    this.retryInterval = (retryInterval + retryJitter).toMillis().toInt()
                    // Ping connection interval is also the ping timeout
                    // During deploys of many pods or Redis node failover, pings may timeout due to heavy traffic on the node
                    // Allow more time than the default of 30s
                    pingConnectionInterval = Duration.ofSeconds(60).toMillis().toInt()
                }
            }

            baseConfig.timeout = Duration.ofSeconds(30).toMillis().toInt()
            codec = LZ4Codec()
            isCheckLockSyncedSlaves = false
        }
)

[mrniko]

This usually happens because you're trying to data stored with different codec

[nigelmenger]

This usually happens because you're trying to data stored with different codec

@mrniko We've been using LZ4Codec for 3 months now. The data that exists in Redis itself was certainly serialized with the same codec. I've confirmed that the version of my service that was previously deployed prior to this one was also using LZ4Codec to serialize the same RBucket.