ClosedChannelException making RSocket request with invalid credentials

[ianbrandt]

Not knowing if this was a Spring Boot, Spring Security, RSocket, or Netty issue, I originally filed this against Spring Boot as spring-projects/spring-boot#34853. They forwarded me here given that exclusively downgrading rsocket-java from 1.1.3 to 1.1.2 corrects the issue.

With Spring Boot 2.6.2 through 2.6.11 (which used rsocket-java 1.1.2) and @EnableRSocketSecurity, if I make a RSocket message mapping request with invalid credentials, I get a RejectedSetupException with message "Invalid Credentials":

https://github.com/ianbrandt/spring-security-samples/blob/a56fcf2148440f2cdddc8e35f8536accc2c4474e/reactive/rsocket/hello-security/src/integTest/java/example/HelloRSocketApplicationITests.java#L79

When I upgrade to 2.6.12, or any later version through 2.7.10 (which use rsocket-java 1.1.3), I get a io.netty.util.IllegalReferenceCountException warning logged, and a ClosedChannelException with null message at the client. Output below. I was expecting the same RejectedSetupException as before 2.6.12 (1.1.3).

Reproducer branch with before and after commits: https://github.com/ianbrandt/spring-security-samples/tree/invalid-credentials-error.

2023-04-04 15:03:55.563  WARN 38792 --- [ctor-http-nio-2] io.netty.util.ReferenceCountUtil         : Failed to release a message: CompositeByteBuf(freed, components=2)

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:147) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101) ~[netty-buffer-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:116) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:306) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:660) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.closeOutboundBufferForShutdown(AbstractChannel.java:672) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:666) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:953) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Expecting actual throwable to be an instance of:
  io.rsocket.exceptions.RejectedSetupException
but was:
  reactor.core.Exceptions$ReactiveException: java.nio.channels.ClosedChannelException
	at reactor.core.Exceptions.propagate(Exceptions.java:396)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
	at reactor.core.publisher.Mono.block(Mono.java:1707)

[ianbrandt]

Merged upstream master into reproducer branch (fixes build by taking up snapshot repo change): https://github.com/ianbrandt/spring-security-samples/tree/invalid-credentials-error.