Tee: on detecting a failed receiver

[OCrylic]

Description
Hi Paolo,

This is a feature request since (from my understanding) it is not implemented in pmacct.

With nfacctd and the plugin tee, we can load-balance the forwarding of the flows to several servers (2 Graylog nodes in my case).
However, if one of the Graylog server crashes, flows are still forwarded to that node since there is no health checks in pmacct.

Health checks is a bit tricky since UDP does not send back acknowledgement and Netflow protocol does not use responses (instead of DNS for example), We have to use another type of health check (REST API to get the status of the graylog cluster or ICMP checks) but using HAProxy or Nginx is not possible since:

• HAProxy does not handle UDP load-balancing
• Nginx OSS is not able to use active checks like REST API or ICMP. this is a Nginx Plus only feature.

Do you have any advice or an other solution for this ? Do you think the health checks could be integrated one day in pmacct ?

[paololucente]

Dear @OCrylic ,

To confirm today there is nothing that could help builtin pmacct. I can think to two possible ways to tackle this:

1. if ICMP is correctly working between the sender and the receiver, it could be intercepted at the sender side so to exclude the failed server from the balancing; obvious question would then be: in this scheme, how to restore the "normal" behavior? Perhaps through a manual map reload? So fail automatically but restore manually, would that be acceptable?

2. If both sender and receiver are pmacct - i am not clear this is your case - since a few releases pmacct does support the use of Redis, a in-memory key-value store, for the purpose of cluster management. So far the only use of Redis has been to register daemons, plugins, etc. with the purpose - one day in future - to act on failing components. How to detect a failing component? Redis does support timed entries. Each pmacct component does register itself at regular time intervals (of course less than the expiry time; currently fixed 60 seconds, not a big deal making it configurable though): if the component stops registering it disappears from the database and hence can be considered failed. This simple mechanism could be used to work out a simple script that checks presence in Redis of the receivers and touch the map at the sender.

Let me know a bit your thoughts on the two options above - and whether you do see something else / different. Then i would slightly re-title this issue and mark it as Enhancement.

Paolo

[OCrylic]

Hi @paololucente,

Regarding the first solution, if you can automatically exclude the failed server at any time during the Netflow replication, why is it not possible to get it back as soon as the ICMP answer (because the ICMP checking would be still made during that time, even if receiver does not answer) ? Unless you are talking about ICMP ping from netflow receiver (Graylog in my case) to netflow sender (pmacct). But maybe the ICMP checks can be made automatically from pmacct to Graylog, every 30 seconds for example, according IP address in tee_receivers.lst and as soon there is a ping fail (or 3 to be sure) the failed server is excluded.

This can be improved I think but it is a good start :) I explain it:
The receiver could answer to ICMP request because it is up and running (communication are OK between sender and receiver, at least from a network point of view) but the application/service on this server receiving and managing the flows could be stopped and yet, the flows will still be sent to that server.
Most of the solutions have now HTTP APIs to provide information about its own status in a json format or else (it is the case for Graylog). The ideal way of checking, instead of an ICMP check, would be pmacct retrieving (with tools like curl) and parsing this json file every 30 seconds and looking for a specific value telling if this receiver is OK or not. But that is more complicated since it completely depends of the solution running on the receiver side.

The second solution you provide does not relate to my case indeed. I am not sure I understood it completely but it could work. If I understand it correctly, instead of using ICMP checks, you relie on the registering process between two pmacct nodes.

[paololucente]

Hi @OCrylic ,

Thanks for your feedback. The first solution would rely on ICMP Unreachables, not ICMP Ping. So the first packet that is sent to a remote UDP port that triggers an ICMP Unreachable back makes that node go off the list. This is why once the node is off, something external should bring it back on once restored. In other words i was not speaking about adding extra ICMP checks to pmacct, ie. ICMP Pings since those would be very coarse-grained, ie. the whole host should go down in order for the recovery mechanism to kick in.

And you are entirely correct on the high-level picture of the second solution i was proposing.

Paolo