Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

org.apache.kafka.clients.NetworkClient Error connecting to node broker:29092 (id: -1 rack: null) java.net.UnknownHostException: broker: Name or service not known #51

Open
dash7ou opened this issue Nov 27, 2020 · 12 comments

Comments

@dash7ou
Copy link

dash7ou commented Nov 27, 2020

control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
rest-proxy | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
rest-proxy | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501070911, tries=1, nextAllowedTryMs=1606501071012) timed out at 1606501070912 after 1 attempt(s)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
ksqldb-server | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
ksqldb-server | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501071531, tries=1, nextAllowedTryMs=1606501071632) timed out at 1606501071532 after 1 attempt(s)
ksqldb-server | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
schema-registry | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
schema-registry | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501072795, tries=1, nextAllowedTryMs=1606501072897) timed out at 1606501072797 after 1 attempt(s)
schema-registry | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker: Name or service not known
control-center | at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
control-center | at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
control-center | at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
control-center | at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
control-center | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
control-center | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501079587, tries=1, nextAllowedTryMs=1606501079698) timed out at 1606501079598 after 1 attempt(s)
control-center | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
rest-proxy | [main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
rest-proxy | java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
rest-proxy | at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149)
rest-proxy | at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)

@dash7ou
Copy link
Author

dash7ou commented Nov 27, 2020

@ybyzek I have an issue when running docker-compose file for cp-all-in-one folder, on ec2 amazon, it runs fine for me locally.

@dash7ou dash7ou changed the title Error connecting to node broker:29092 (id: -1 rack: null) org.apache.kafka.clients.NetworkClient Error connecting to node broker:29092 (id: -1 rack: null) java.net.UnknownHostException: broker: Name or service not known Nov 27, 2020
@cosbor11
Copy link

cosbor11 commented Jan 5, 2021

This happens for me too

@konfortes
Copy link

Same here for local run.
osx 10.15.5
Docker 3.0.3(51017)

@wscardua
Copy link

wscardua commented Jan 14, 2021

me too

osx 10.15.7
docker 3.0.4 (51218)

@ybyzek
Copy link
Contributor

ybyzek commented Jan 15, 2021

Things to check:

  1. Is the broker up? (docker-compose ps)
  2. Can the other container ping the broker container? (docker-compose exec control-center ping broker)
  3. Are there any errors in the logs (search through docker-compose logs)
  4. Which branch of this repo is checked out? Make sure to run latest (latest or 6.0.1-post)

@mhbrk
Copy link

mhbrk commented Jan 17, 2021

docker-compose logs -f broker

Attaching to broker
broker | ===> User
broker | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
broker | ===> Configuring ...
broker | ===> Running preflight checks ...
broker | ===> Check if /var/lib/kafka/data is writable ...
broker | ===> Check if Zookeeper is healthy ...
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=broker
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=11.0.9.1
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu11-ca
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/share/java/cp-base-new/paranamer-2.8.jar:/usr/share/java/cp-base-new/netty-resolver-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-library-2.13.2.jar:/usr/share/java/cp-base-new/metrics-core-2.2.0.jar:/usr/share/java/cp-base-new/audience-annotations-0.5.0.jar:/usr/share/java/cp-base-new/kafka-clients-6.0.1-ccs.jar:/usr/share/java/cp-base-new/commons-cli-1.4.jar:/usr/share/java/cp-base-new/slf4j-api-1.7.30.jar:/usr/share/java/cp-base-new/jopt-simple-5.0.4.jar:/usr/share/java/cp-base-new/jackson-databind-2.10.5.jar:/usr/share/java/cp-base-new/netty-transport-4.1.48.Final.jar:/usr/share/java/cp-base-new/jackson-module-scala_2.13-2.10.5.jar:/usr/share/java/cp-base-new/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-logging_2.13-3.9.2.jar:/usr/share/java/cp-base-new/jackson-dataformat-yaml-2.10.5.jar:/usr/share/java/cp-base-new/argparse4j-0.7.0.jar:/usr/share/java/cp-base-new/gson-2.8.6.jar:/usr/share/java/cp-base-new/scala-collection-compat_2.13-2.1.6.jar:/usr/share/java/cp-base-new/jackson-datatype-jdk8-2.10.5.jar:/usr/share/java/cp-base-new/jackson-annotations-2.10.5.jar:/usr/share/java/cp-base-new/utility-belt-6.0.1.jar:/usr/share/java/cp-base-new/netty-buffer-4.1.48.Final.jar:/usr/share/java/cp-base-new/common-utils-6.0.1.jar:/usr/share/java/cp-base-new/scala-java8-compat_2.13-0.9.1.jar:/usr/share/java/cp-base-new/zookeeper-jute-3.5.8.jar:/usr/share/java/cp-base-new/zookeeper-3.5.8.jar:/usr/share/java/cp-base-new/kafka_2.13-6.0.1-ccs.jar:/usr/share/java/cp-base-new/zstd-jni-1.4.4-7.jar:/usr/share/java/cp-base-new/jackson-core-2.10.5.jar:/usr/share/java/cp-base-new/lz4-java-1.7.1.jar:/usr/share/java/cp-base-new/netty-handler-4.1.48.Final.jar:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar:/usr/share/java/cp-base-new/jackson-module-paranamer-2.10.5.jar:/usr/share/java/cp-base-new/snappy-java-1.1.7.3.jar:/usr/share/java/cp-base-new/netty-common-4.1.48.Final.jar:/usr/share/java/cp-base-new/jackson-dataformat-csv-2.10.5.jar:/usr/share/java/cp-base-new/snakeyaml-1.26.jar:/usr/share/java/cp-base-new/netty-transport-native-epoll-4.1.48.Final.jar:/usr/share/java/cp-base-new/netty-codec-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-reflect-2.13.2.jar
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.19.121-linuxkit
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=appuser
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/appuser
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/appuser
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=120MB
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1992MB
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=126MB
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@a7e666
broker | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
broker | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
broker | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.24.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.24.0.2:2181: Connection refused
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.24.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.24.0.3:59678, server: zookeeper/172.24.0.2:2181
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.24.0.2:2181, sessionid = 0x1000024ccb10000, negotiated timeout = 40000
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000024ccb10000 closed
broker | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000024ccb10000
broker | ===> Launching ...
broker | ===> Launching kafka ...
broker | [2021-01-17 21:17:02,315] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
broker | [2021-01-17 21:17:03,638] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
broker | [2021-01-17 21:17:03,679] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092
broker | at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:260)
broker | at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:271)
broker | at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:2297)
broker | at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2376)
broker | at kafka.server.KafkaConfig.(KafkaConfig.scala:2354)
broker | at kafka.server.KafkaConfig.(KafkaConfig.scala:1807)
broker | at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
broker | at kafka.Kafka$.main(Kafka.scala:68)
broker | at kafka.Kafka.main(Kafka.scala)
broker exited with code 1

@mhbrk
Copy link

mhbrk commented Jan 17, 2021

also it doesn't have - "29092:29092" in this part: https://github.com/confluentinc/cp-all-in-one/blob/6.0.1-post/cp-all-in-one/docker-compose.yml#L20

@ybyzek
Copy link
Contributor

ybyzek commented Jan 18, 2021

@mhbrk from the error logs:

broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092

It looks like 9092 is reused twice, this is not allowed as described in the message. So it appears your local docker-compose.yml file has been modified. Please update it to match what is in GitHub (which is validated to work) and try again.

@DavoBR
Copy link

DavoBR commented Feb 17, 2021

about the port 29092, i am facing with the following error

image

container console output:

confluentinc_cp-server_6.1.0.txt

@OneCricketeer
Copy link

OneCricketeer commented Mar 29, 2021

also it doesn't have - "29092:29092" in this part

broker hostname isn't resolvable by the host, anyway, so a port forward is not the correct location to debug the problem

As the logs say, your listeners config cannot use the same port. Without this, your broker dies as well as all other services, such as control center, that depend on it

You may explicitly define KAFKA_LISTENERS, otherwise the default behavior in the Docker image is to use the advertised listeners, and replace the host names with 0.0.0.0

These get printed in the logs when the container starts https://github.com/confluentinc/confluent-docker-utils/blob/master/confluent/docker_utils/cub.py#L469

@tooptoop4
Copy link

@dash7ou did u solve?

@nicolepastrana
Copy link

This is what worked for me in my M1 Chip:

https://github.com/confluentinc/cp-all-in-one/blob/7.1.1-post/cp-all-in-one/docker-compose.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants