I can't connect to Kafka in OS X

[inancgumus]

Hi.

I'd been having problems launching Kafka (more about this in this recent thread: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/confluent-platform/DRd_XfeW0C8).

Now, I launched Kafka, however, I can't connect it:

# the kafka container uses this port
$ telnet localhost 29092
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host

PS: I'm trying this docker-compose.yml.

This is my altered docker-compose.yml file:

I tried to expose the ports, however, I think, in network host mode, it can't be done.

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    network_mode: host
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

  kafka:
    image: confluentinc/cp-kafka:latest
    network_mode: host
    ports:
      - "29092:29092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: localhost:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
    extra_hosts:
      - "moby:127.0.0.1"

[inancgumus]

You might read this issue, there are some enlightnings. Thx to @mal . It solves many of the issues by using an external network tap. So, the containers don't need network_mode: host and extra_hosts configs.

[inancgumus]

Probably, noone is looking at these issues...

[tmendenhall]

I was able to have some success on macOS (docker community version) using the following compose file:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.2.1
    ports:
     - 32181:32181
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"
      - "localhost: 127.0.0.1"

  kafka:
    image: confluentinc/cp-kafka:3.2.1
    ports:
     - 29092:29092
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_DELETE_TOPIC_ENABLE: "true"
    extra_hosts:
      - "moby:127.0.0.1"
      - "localhost: 127.0.0.1"

Which allows kafka to connect to the zookeeper instance using the bridge network but both remain accessible to a Java application running on my development machine at localhost and the exposed correct port. I hope this proves useful.

Note: I am not using the most recent version of the cp-zookeeper and cp-kafka docker images.

[inancgumus]

@tmendenhall Thank you for the help. I'm no longer using these services as they're hard to configure and doesn't have enough support for non-enterprise customers.

I switched to Golang and its ecosystem which is more welcoming and even has a kafka in go alternative which simpler to use but coming with less feature set of course, however enough for the most.

Btw, you can look at this thread as well, there're some developments for Docker network mode support in OS X (after some pressure from the community to the Docker team. As they're weren't dealing with the issue although they say the otherwise.).

[santthosh]

I have the latest steps written down here https://github.com/santthosh/kafka-for-mac with the latest version of cp-zookeeper and cp-kafka docker images

[OneCricketeer]

The solution isn't via the extra_hosts, it's actually adding more listeners

https://rmoff.net/2018/08/02/kafka-listeners-explained/

[vbyno]

@Cricket007 , did you manage to make it work? Could you please share your docker-compose.yml file?

[OneCricketeer]

That article explains exactly how to make it work on a Mac, as well as links to the all-in-one compose file of this repo.

Note: there are two ports. 9092 and 29092

[elbarri]

This one solved it for me:
https://stackoverflow.com/questions/64805874/problem-with-advertised-listener-on-macos

It should include KAFKA_LISTENER_SECURITY_PROTOCOL_MAP and KAFKA_INTER_BROKER_LISTENER_NAME