New Homer 7 docker installed - no data shown for HEPv3 traffic coming from opensips 2.4

[solarmon]

Hi,

I have a new Homer 7 installed using the docker method as per:

https://github.com/sipcapture/homer/wiki/Quick-Install#-docker-install

This is on a Centos 7 machine:

CentOS Linux release 7.8.2003 (Core)

I can see that docker is running the Homer containers:

# docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                       PORTS                                                                                            NAMES
9d9acf0c534a        sipcapture/webapp                  "/docker-entrypoin..."   22 hours ago        Up 22 hours                                                                                                                   homer-webapp
93aa7e4a7465        sipcapture/heplify-server:master   "./heplify-server"       22 hours ago        Up 22 hours                  0.0.0.0:9060->9060/tcp, 0.0.0.0:9060->9060/udp, 9090/tcp                                         heplify-server
9b05fc89c980        prom/alertmanager:latest           "/bin/alertmanager..."   22 hours ago        Restarting (1) 8 hours ago                                                                                                    alertmanager
983d50a1b540        prom/node-exporter:latest          "/bin/node_exporte..."   22 hours ago        Up 22 hours                  9100/tcp                                                                                         nodeexporter
5cd97de12b6a        postgres:11-alpine                 "docker-entrypoint..."   22 hours ago        Up 22 hours (healthy)        5432/tcp                                                                                         db
a7d9cfe6cc99        grafana/loki:master                "/usr/bin/loki -co..."   22 hours ago        Up 22 hours                  0.0.0.0:3100->3100/tcp                                                                           loki
5a32d841fcdf        grafana/grafana:master             "/run.sh"                22 hours ago        Up 22 hours (healthy)        3000/tcp                                                                                         grafana
75d1dcd8f4d9        prom/prometheus:latest             "/bin/prometheus -..."   22 hours ago        Restarting (1) 8 hours ago                                                                                                    prometheus
e706c611de43        stefanprodan/caddy                 "/sbin/tini -- cad..."   22 hours ago        Up 22 hours                  0.0.0.0:3000->3000/tcp, 0.0.0.0:9080->9080/tcp, 0.0.0.0:9090->9090/tcp, 0.0.0.0:9093->9093/tcp   caddy

I have set up opensips 2.4 to send HEPv3 traffic to port 9060 of the Homer server and tshark captures shows that it is reaching the Homer server.

However, in the Homer web GUI it does not show any data, and showing the 'Loading...' message:

Please can you advise on how to troubleshoot this, as I don't have any experience with Homer and Docker.

Thank you!

[lmangani]

Most likely the ports are locked by a firewall. Please doublecheck traffic can actually reach containers as a first step.

[lmangani]

You can confirm this by running hepgen against your HOMER setup

cd /usr/src
git clone https://github.com/sipcapture/hepgen.js
cd hepgen.js 
npm install

The execute replacing 127.0.0.1 with your HOMER/Heplify-Server IP

node hepgen.js -s 127.0.0.1 -p 9060 -c "./config/b2bcall_rtcp.js"

If this doesn't work, definitely traffic is not hitting the containers.

[solarmon]

@lmangani

Thank you for the response.

I did check iptables and there is a DOCKER chain that has the ports opened:

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:9093
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:9090
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:9080
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:3000
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.4           tcp dpt:3100
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.8           tcp dpt:9060
ACCEPT     udp  --  0.0.0.0/0            172.18.0.8           udp dpt:9060

Although, the IP address is not of the VM interface that the docker contains are on. This seems to be the internal IP addresses that the Docker containers uses - so I don't know whether these actually gets matched. I assume this DOCKER chain was created as part of the Homer docker-compose install, or by Docker itself?

I tried the hepgen test, on the same machine as where Homer docker installs, and I still do not get any data. I assume it should appear immediately if it is working.

I will now check a few other things, now that I have test that can be used, like turning off iptables just to test.

Thank you

[lmangani]

I guess you need to explicitly open the ports on your OS level firewall:

firewall-cmd --zone=public --add-port=9060/udp --permanent
firewall-cmd --reload

[solarmon]

@lmangani

I think the hepgen test I did didn't appear in Homer because I didn't 'refresh' it?

After I turned off iptables and I ran hepgen again. And again the traffic did not appear.

After I logged out of the Homer web app and logged back in, I could see the traffic. And there were traffic from the two timestamps for when I did the hepgen tests.

I enabled iptables again and ran hepgen again. This time the test traffic appeared straight away.

So I'm not sure why the first hepgen test traffic didn't appear straight away.

I checked 'iptables -L' and the DOCKER chain is not present, so I assume this gets added when docker starts the containers?

I will try the test traffic from opensips without iptables running.

[solarmon]

@lmangani

I've kept the firewalld service off for now until I get something working.

I installed hepgen on the opensips node that I trying to send HEPv3 traffic from. Homer shows the test traffic when I run this hepgen script on that opensips node.

So I think the issue is with the format of the HEPv3 traffic coming from opensips.

I have installed the HEP dissector for Wireshark to look at the capture traffic and it decodes it correctly as HEPv3.

One thing to note is that I am not using TLS to encrypt the traffic. I assume this is not an issue for Homer to receive unencrypted HEPv3 traffic?

Here is the HEPv3 dissected packet coming from opensips (this is just a SIP OPTIONS example) - I have sanitised any sensitive info:

Does that look OK as a HEPv3 packet?

Why would this not be ingested by Homer/heplify, but the hepgen traffic does get ingested?

[lmangani]

If the packet is decoded, then its valid HEP. The issue must be elsewhere since hepgen.js is able to correctly insert the data. There's no TLS in HEP by default, so that's not an issue either.

Perhaps you're not sending Calls and are looking into the wrong transaction type?

[solarmon]

That example was just for the first SIP packet in the trace/capture. There is also an INVITE for the test call I did - which is dissected as HEPv3. But it shouldn't matter what type of SIP packet is sent, no? They should all be ingested if it is HEPv3?

Is there any logs I can check in the heplify-server docker container to see if there are any issues with accepting/ingesting the HEPv3 traffic?

[solarmon]

Comparing the captures between opensips and hepgen packets, Wireshark does highlight an warning with the opensips packets - with the error message about "Trailing stray characters":

Would this be the reason why it is not being ingested?

How would I find out what/where these trailing stray characters are? Maybe a question for the opensips mailing list too.

[solarmon]

OK, looking into the HEPv3 captures coming from opensips, I see that there is 'trailing' data at the end. For example, for the INVITE, there is the following:

.....70110F001-0000FA9B-5EEA5D1C-0000000E@a.b.c.d

Which is the same as the value for the Call-ID header.

I need to check with the opensips mailing list to ask why this is happening and whether it is normal.

[solarmon]

Hi @lmangani

opensips mailing list still not responded to my question.

Meanwhile, can you confirm that such trailing characters in the HEPv3 packet will cause it to not be injected.

Thank you.

[lmangani]

@solarmon we don't have the packets so we cannot confirm anything, sorry. This being said, If Wireshark says its malformed, its malformed.

[solarmon]

@lmangani

Sorry, I meant whether there are any checks for such malformed packets that would prevent it from being accepted.

Yes, I'm seeing it in Wireshark and the HEPv3 dissector is giving this warning, but I would like to understand whether Homer/Heplify has such checks and rejects such malformed packets. Then I know that this is the cause of the packets not being accepted.

[lmangani]

@solarmon absolutely you can assume so. Only valid HEP would be decoded by HEP servers.

[negbie]

@solarmon check logs with sudo docker logs -f --tail 100 heplify-server

[solarmon]

Hi @negbie

Thank you for that tip!

The logs don't seem to generate anything specific for when I send opensips and hepgen HEPv3 traffic to it. However, I do get these logs entries constantly:

2020/06/22 11:48:55.127071 tls.go:62: INFO new TLS connection 172.18.0.1:36606 -> 172.18.0.8:9060
2020/06/22 11:48:55.128539 tls.go:88: WARN tls: first record does not look like a TLS handshake from 172.18.0.1:36606

What does this mean?

172.18.0.8 is the heplify-server container and I don't know what 172.18.0.1 is, as it is not any of the container IP addresses.

[lmangani]

@solarmon that's the subnet route for your Docker containers

[solarmon]

@solarmon that's the subnet route for your Docker containers

@lmangani Sorry, I don't understand what you mean by that? The logs show that this TLS connection is coming from 172.18.0.1:36606 - so it is coming from a host/object that has this IP address, which I can't identify at the moment.

What do you mean by 'subnet route'?

[solarmon]

@lmangani

I've been comparing the hepgen packets and the opensips sipcapture packets and it seems there is a difference , which relates to this Call-ID value that I'm seeing.

In the hepgen HEPv3 packet, the "Correlation ID" section, which seems to also have the Call-ID value for it, is BEFORE the "Encapsulated Payload [truncated]" section.

In the opensips sipcature HEPv3 packet I see that the "Correlation ID" is AFTER "Encapsulated" Payload [truncated]", so appears after the encapsulated SIP packet.

I think this is the reason why the Wireshark HEPv3 dissector is warning and showing this Call-ID / Correlation ID after the SIP packet; and possibly why Homer/Heplify is rejecting it because it is malformed?

[solarmon]

@lmangani

I think this "Trailing stray characters" and the "Correlation ID" and "Encapsulated" Payload [truncated]" ordering is a red herring, although I'd like to understand why the HEP dissector for Wireshark is giving such a warning.

I was able to use sngrep to send captured SIP packets on the opensips server and send it to my Homer/Heplify server and it appeared OK.

I noticed that both hepgen and sngrep were send HEPv3 packets as UDP and not TCP. I changed opensips to send HEPv3 packets in UDP and it work!

I thought the Homer docker setup was listening on both TCP and UDP port 9060, which is what is suggested by netstat:

# docker exec -it heplify-server netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:44651        0.0.0.0:*               LISTEN      -
tcp        0      0 172.18.0.8:43330        172.18.0.4:3100         ESTABLISHED 1/heplify-server
tcp        0      0 172.18.0.8:45912        172.18.0.6:5432         ESTABLISHED 1/heplify-server
tcp        0      0 :::9096                 :::*                    LISTEN      1/heplify-server
tcp        0      0 :::9060                 :::*                    LISTEN      1/heplify-server
udp        0      0 127.0.0.11:59765        0.0.0.0:*                           -
udp        0      0 :::9060                 :::*                                1/heplify-server
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path

So why is it not accepting HEPv3 TCP packets, but is accepting HEPv3 UDP packets?

[solarmon]

Hi @lmangani

Any advice on how to resolve this, and why TCP does not seem to work?