No data written to Kafka topic

[saloni-vunet]

I am testing vFlow in Docker with Confluent Kafka. Creating a test netflow v5 flow with nerdalert/nflow-generator. The topic is created in Kafka but no data is written to the topic.
I see the following error in the logs:

kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused

I am running vFlow with the command
docker run -d -p 4739:4739 -p 4729:4729 -p 6343:6343 -p 9996:9996 -e VFLOW_KAFKA_BROKERS="172.21.0.3:9092" mehrdadrad/vflow

[mehrdadrad]

interesting! can you send me the full log?

[saloni-vunet]

vFlow running with PID 9.
[vflow] 2020/12/01 13:48:46 open /etc/vflow/vflow.conf: no such file or directory
[vflow] 2020/12/01 13:48:46 Welcome to vFlow v.0.7.0 Apache License 2.0
[vflow] 2020/12/01 13:48:46 Copyright (C) 2018 Verizon. github.com/VerizonDigital/vflow
[vflow] 2020/12/01 13:48:47 starting prometheus http server ...
[vflow] 2020/12/01 13:48:47 ipfix is running (UDP: listening on [::]:4739 workers#: 200)
[vflow] 2020/12/01 13:48:47 sFlow is running (UDP: listening on [::]:6343 workers#: 200)
[vflow] 2020/12/01 13:48:47 ipfix RPC enabled
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.sflow
[vflow] 2020/12/01 13:48:47 netflow v9 is running (UDP: listening on [::]:4729 workers#: 200)
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 netflow v5 is running (UDP: listening on [::]:9996 workers#: 200)
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.ipfix
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.netflow9
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.netflow5
[vflow] 2020/12/01 13:49:58 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:09 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:15 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:15 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:15 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:16 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:21 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:22 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:22 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:24 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:24 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:25 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:26 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:27 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:27 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:28 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:29 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:30 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:31 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:32 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused

This is the log for vFlow container.

I modified the run command and added the --network option to run vFlow in the same overlay network of Confluent Kafka, but it still results in the exact same error log.

[mehrdadrad]

can you try w/ docker-compose?

docker-compose up

[saloni-vunet]

I tried with docker-compose but still the same error. I believe it is just picking the default broker for producer and not the specified configuration

vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/vflow.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 Welcome to vFlow v.0.7.0 Apache License 2.0
vflow_1       | [vflow] 2020/12/02 07:43:51 Copyright (C) 2018 Verizon. github.com/VerizonDigital/vflow
vflow_1       | [vflow] 2020/12/02 07:43:51 starting prometheus http server ...
vflow_1       | [vflow] 2020/12/02 07:43:51 ipfix is running (UDP: listening on [::]:4739 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 netflow v5 is running (UDP: listening on [::]:9996 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 sFlow is running (UDP: listening on [::]:6343 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 netflow v9 is running (UDP: listening on [::]:4729 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 ipfix RPC enabled
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.sflow
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.netflow5
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.netflow9
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.ipfix
vflow_1       | [vflow] 2020/12/02 07:52:33 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:35 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:35 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:37 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:37 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:38 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:38 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:40 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:41 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:41 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:43 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:44 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused