Using Keepalive for TCP Probe?

[JamesYYang]

I think close tcp connection immediately after connection Established is not good design. This will case a lot of socket in TIME_WAIT status.

So I think tcp probe can keep the connection open until server timeout or error occurs. And report the fail if re-connection failed.

What do you think?

[proditis]

Hi @JamesYYang 👋 ,
I can definitely see benefits in adding keepalive support to tcp probes, that is a good idea.

Some things that we may need to consider

• Do we consider a probe as failed for having its connection closed outside of probe intervals? What happens if a remote host, closes the connection sooner than our keep-alive interval? (eg the remote end has idle-timeout<=keepalive-interval)
• Do we convert the "probe" interval into keepalive-interval? Or are we going to keep trak of 2 different timers?
• What other flags we may need to be aware or modify for our case? Do we also modify/use TCP_KEEPCNT or leave these up to the underlying OS?

@haoel I think we should have this as a configurable option to enable keepalive for a tcp probe. But i dont have the slightest idea about how complicated its implementation may be.

[JamesYYang]

Hi @proditis

I think we can do it in a simple way.

• Probe maintain a status: success or failure. Report the status when probe timer trigger.
• Probe start a goroutine to keep trying to connect target host.
• When TCP connection become Established change status to success.
• When TCP connection closed no matter what reason (timeout or server close) just retry connect in back off way. If failure count > threshold, change status to failure.

[haoel]

@JamesYYang thanks for this consideration.

I have the following thoughts.

EaseProbe's Purpose

I think the keepalive is good for reusing the tcp connection, however, I don't think this is suitable for EaseProbe.

EaseProbe not just only probes the service, but also probes the network, which means we need to know the latency of the DNS, TCP connection, TLS negotiation, and so on, we need to check whether we still can establish a new connection to the server, otherwise, the server should be marked to down. Besides, we have round trip time that needs to be calculated, which could be a factor for the healthy in future.

So, every time EaseProbe must start over to do the fresh probe work, without this, the EaseProbe is functionally broken.

is TIME_WAIT a problem?

TIME_WAIT needs 2MSL, I think it should be fine. on Linux, it is 60s, on Mac, it is 15s. The default probe interval is 60s. I don't think TIME_WAIT could be a problem.

What's the good design?

Different people could have different answers. Different situations also could have different answers.

This is a complicated topic, but I quite like two principles - 1) KISS 2) Benefit-Cost ratio

TCP keepalive would introduce complexity and fewer benefits. So, the benefit-cost ratio is not good.

Let's just keep it simple and stupid.

[proditis]

Thank you @haoel for the thorough explanation.
Yes I agree with you 100%, it gets complicated for no real benefits other than maybe some extremely large setups.

[JamesYYang]

@haoel thanks for sharing your thoughts about this topic.

I also 100% agree with your opinion about good design.

Actually we have similar probe program in our company. It detect the API endpoint health for our API gateway. Because it is internal network, so we can ignore latency of DNS and TLS negotiation and so on. And the probe interval default is 5s. There are thousands of target endpoints we need detect.

So this is why I choose reuse tcp connection. If creating a new connection every time, there will be too many connections need to be established at same time.

So, yes, Different situations could have different answers . I hope my explanation can let you know my problem, if you have other thoughts, please sharing with me.

[haoel]

@JamesYYang

Actually, the DNS, TCP, TLS probe should NOT be ignored by the probe. I've seen many DNS, TCP, and TLS network issues in many companies.

• The DNS is not as stable as you think, especially since there are thousands of endpoints as your mentioned. And sometimes, someone would mistakenly set the wrong IP in DNS server.
• TCP connection also could be failed in some cases, one example is if the server is full of the TCP connection you cannot establish a new TCP connection.
• TLS negotiation also could have problems, because the certification could be expired.

As you probe your service so frequently, I believe you try to find all of the failures, so, you shouldn't ignore the DNS, TCP, or TLS failures.

And using the keepalive to reuse the connection, not only just lose the integral network probe functionality, but also introduce the state for your service, and you have to manage many situations which make things much more complicated.

if your X-Problem just tries to solve the issue of TIME_WAIT. I think you should go the better way instead of using keepalive.

I have several solutions for you to solve the X-Problem.

0. make the MSL shorter, for example: echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
1. enable the tcp_tw_reuse and tcp_tw_recycle, these two options are dangerous for a service, but it is fine for the client.
2. using the docker has a private IP. one IP could have 65535 TCP port, if the probe interval is 5s, then you can have 65535 / (60/5) = 5400 probers. (60s is the Linux 2MSL) , you can have multiple docker containers...

[JamesYYang]

ok, got it, thanks @haoel

[haoel]

@JamesYYang PR #159 would address the TIME_WAIT issue completely. Please take a look.

[JamesYYang]

I think SetLinger can solve my problem, thanks for sharing.

[cuishuang]

再看时物是人非...😞