Eureka Server + Spring Boot Admin throwing org.springframework.web.reactive.function.client.WebClientRequestException

[luanvelasco]

Spring Boot Admin Server information

• Version:3.2.3

• Spring Boot version:3.2.5

• Configured Security:No auth

Client information

• Spring Boot versions:3.2.5

• Used discovery mechanism:Eureka

• Webflux or Servlet application:

Description

I'm setting up a Eureka server along with Spring Boot Admin to monitor the applications.

For this particular project, I'm using:

Java 21
Spring Boot 3.2.5
Spring Cloud 2023.0.1
Spring Boot Admin 3.2.3

My application "luan-service-registry" registers normally with my Eureka server. However, in Spring Boot Admin, for health check, the error message appears:

exception -> org.springframework.web.reactive.function.client.WebClientRequestException
message - > Failed to resolve 'W11BGRC156.luan.com.com' [A(1), AAAA(28)] after 7 queries

This is the console log:

`2024-05-07T18:44:35.033-03:00 INFO 25000 --- [luan-admin] [ctor-http-nio-1] d.c.b.a.server.services.StatusUpdater : Couldn't retrieve status for Instance(id=f8c68ca6e855, version=0, registration=Registration(name=LUAN-ADMIN, managementUrl=http://W11BGRC156.luan.com.com:8888/actuator, healthUrl=http://W11BGRC156.luan.com:8888/actuator/health, serviceUrl=http://W11BGRC156.luan.com:8888, source=discovery), registered=true, statusInfo=StatusInfo(status=UNKNOWN, details={}), statusTimestamp=2024-05-07T21:44:14.538480400Z, info=Info(values={}), endpoints=Endpoints(endpoints={health=Endpoint(id=health, url=http://W11BGRC156.luan.com:8888/actuator/health)}), buildVersion=null, tags=Tags(values={}))

org.springframework.web.reactive.function.client.WebClientRequestException: Failed to resolve 'W11BGRC156.luan.com' [A(1), AAAA(28)] after 7 queries
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136) ~[spring-webflux-6.1.6.jar:6.1.6]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ Request to GET health [DefaultWebClient]
Original Stack Trace:
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136) ~[spring-webflux-6.1.6.jar:6.1.6]
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.Mono.subscribe(Mono.java:4568) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:205) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:229) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:279) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.maybeOnError(FluxConcatMapNoPrefetch.java:327) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:212) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.SinkManyEmitterProcessor.drain(SinkManyEmitterProcessor.java:476) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.SinkManyEmitterProcessor$EmitterInner.drainParent(SinkManyEmitterProcessor.java:620) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxPublish$PubSubInner.request(FluxPublish.java:874) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:337) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.Operators$DeferredSubscription.request(Operators.java:1743) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:196) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:205) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:311) ~[reactor-netty-http-1.1.18.jar:1.1.18]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:205) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:172) ~[reactor-netty-core-1.1.18.jar:1.1.18]
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:488) ~[reactor-netty-core-1.1.18.jar:1.1.18]
at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:436) ~[reactor-netty-core-1.1.18.jar:1.1.18]
at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:205) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:583) ~[reactor-netty-core-1.1.18.jar:1.1.18]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.Operators.error(Operators.java:198) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:53) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.Mono.subscribe(Mono.java:4568) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.6.5.jar:3.6.5]
at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:576) ~[reactor-netty-core-1.1.18.jar:1.1.18]
at reactor.netty.transport.TransportConnector.lambda$doResolveAndConnect$11(TransportConnector.java:375) ~[reactor-netty-core-1.1.18.jar:1.1.18]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86) ~[netty-resolver-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1159) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1098) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:457) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:69) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:526) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:380) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsQueryContext$5.run(DnsQueryContext.java:315) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: java.net.UnknownHostException: Failed to resolve 'W11BGRC156.luan.com' [A(1), AAAA(28)] after 7 queries
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1151) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1098) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:457) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:69) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:526) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:380) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.resolver.dns.DnsQueryContext$5.run(DnsQueryContext.java:315) ~[netty-resolver-dns-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: io.netty.resolver.dns.DnsNameResolverTimeoutException: [3155: /181.213.132.3:53] DefaultDnsQuestion(W11BGRC156.luan.com. IN A) query '3155' via UDP timed out after 5000 milliseconds (no stack trace available)
`
Services:
luan-services.zip

[SteKoe]

Hi @luanvelasco,

it seems that the given domain is not resolved via DNS (DnsNameResolverTimeoutException). Please try to ping or trace the given url http://w11bgrc156.luan.com.com/ from the system where Spring Boot Admin is running. Right now this does not look like a SBA issue but related to your local config of systems and DNS.

Regards,
SteKoe

[luanvelasco]

Hi @SteKoe

Right. It really seems to be a problem with DNS. By inserting this dependency below. Locally it works without any problems. However, my clients and servers are located in a PCF private cloud foundry environment. Even adding this dependency doesn't work.

org.springframework.boot spring-boot-starter-webflux io.projectreactor.netty reactor-netty-http

Currently, with older versions it works without problems.

Spring Cloud 2021.0.1
Java 17
Spring boot 2.7.18

For this new version. Is there any specific configuration that must be done?

[erikpetzold]

Hi @luanvelasco

• you wrote that it is working locally, so in general there shouldn't be bug or something like that, only a configuration issue
• the reactive web client is already used in your first post, as you can see in the stacktrace, so should not be influenced by new dependencies
• you updated many things at once... the problem could come from any of these. I would recomment to
  • check the changelog of spring cloud
  • maybe use java 17 to make sure the problem does not come from java update
  • check if there are any properties or imports that need to be changed from javax to jakarta
  • as the dns resolving seems to be done by netty (see stacktrace), check the chngelog and configuration there
• are you using the exact same cloud foundry enviroment as with the old version or has something changed there? Do you need to add some configuration?

As @SteKoe already said, I do not see any connection to Spring Boot Admin at the moment