Random outbound connection timeouts based on server load #587

mavci · 2022-06-23T20:32:40Z

Hello,

We've been experiencing random outbound connection timeouts based on server load for a very long time. After restarting the server, the problems go away, but after a while the timeouts start again. After some research I found these issues related to this topic:

docker/for-win#8861
docker/for-mac#3448
docker/for-mac#6086
docker/for-win#12671
docker/for-win#12761

I didn't put the technical logs here because these issues contain the relevant logs and results that I already have.
Anyone else having similar issues? And how can we fix it, any suggestions?

Thank you.

rossinineto · 2022-06-25T12:06:06Z

I reported in https://github.com/docker/for-win/issues/8861

Spenhouet · 2022-06-28T07:27:19Z

@djs55 Contrary to the title of this issue, this is not actually random at all, is very reproducible and has nothing to do with server load. Because of this our application no longer works with any version above docker desktop 4.5.0 on Mac and 4.5.1 on Windows. We now have to force all customers to downgrade. This is rather serious for us. We will share a reproduction soon.

EDIT: We might open a new issue since this one is so unspecific and off from the actual issue.

jdeitrick80 · 2022-09-05T12:13:23Z

I have also seen this issue in versions >4.5.1 including the latest version, but have found that it can also be triggered with low amounts of traffic. The following is how I have been able to reproduce the issue.

docker run --name session-test -it -v /mnt/c/Users/jde/sessions/test:/test python:buster bash
root@17c8b33e70e6:/# pip install --quiet requests
root@17c8b33e70e6:/# cd test/
root@17c8b33e70e6:/test# python sessions.py
10:52:18: request 1
Request complete, sleep 30
10:52:50: request 2
Request complete, sleep 120
10:54:51: request 3
Request complete, sleep 420
11:01:51: request 4
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

sessions.py

import requests
from datetime import datetime
import time

s = requests.Session()

steps = [30, 120, 420, 420]
step = 1
for i in steps:
    print(datetime.now().strftime("%H:%M:%S") + ": request " + str(step))
    r3 = s.get('https://wttr.in')
    print("Request complete, sleep " + str(i))
    step+=1
    time.sleep(i)

As has been mentioned before if I look at a trace from the containers point of view I only see TCP SYNs being sent out during the 4th attempt after waiting 420s since the last request. Also if I kill the vpnkit while it is still trying the 4th attempt then when the vpnkit starts back up the 4th requests is able to complete successfully.

Some things that I have noticed that I do not think were previously mentioned. If I look at a trace from the host I see the TCP SYNs going out and TCP SYN ACKs coming back from the server, but these are not passed on to the container. If I start up another container while the first is trying unsuccessfully to do the 4th attempt it also is not able to reach the same destination, but is able to reach other destinations.

docker run -it python:buster bash
root@14437db6e250:/# curl https://wttr.in
curl: (7) Failed to connect to wttr.in port 443: Connection timed out
root@14437db6e250:/# curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
root@14437db6e250:/# curl https://wttr.in
curl: (7) Failed to connect to wttr.in port 443: Connection timed out

The cause of the issue seems to have something to do with using sessions and having a client side keep alive interval being >=60s. If I change to a 30s client keep alive interval I do not run into the issue.

docker run --name session-test -it -v /mnt/c/Users/jde/sessions/test:/test python:buster bash
root@425db0e6590a:/# pip install --quiet requests
root@425db0e6590a:/# cd test/
root@425db0e6590a:/test# python sessions-ka30.py
11:41:35: request 1
Request complete, sleep 30
11:42:06: request 2
Request complete, sleep 120
11:44:06: request 3
Request complete, sleep 420
11:51:06: request 4
Request complete, sleep 420
root@425db0e6590a:/test#

sessions-ka30.py

import requests
from datetime import datetime
import time

import socket
from requests.adapters import HTTPAdapter

class HTTPAdapterWithSocketOptions(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        self.socket_options = kwargs.pop("socket_options", None)
        super(HTTPAdapterWithSocketOptions, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        if self.socket_options is not None:
            kwargs["socket_options"] = self.socket_options
        super(HTTPAdapterWithSocketOptions, self).init_poolmanager(*args, **kwargs)

KEEPALIVE_INTERVAL = 30
adapter = HTTPAdapterWithSocketOptions(socket_options=[(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, KEEPALIVE_INTERVAL), (socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, KEEPALIVE_INTERVAL)])
s = requests.Session()
s.mount("http://", adapter)
s.mount("https://", adapter)

steps = [30, 120, 420, 420]
step = 1
for i in steps:
    print(datetime.now().strftime("%H:%M:%S") + ": request " + str(step))
    r3 = s.get('https://wttr.in')
    print("Request complete, sleep " + str(i))
    step+=1
    time.sleep(i)

I hope this information helps in resolving the issue or provides a work around for others experiencing it.

I have also added this information to https://github.com/docker/for-win/issues/8861

rossinineto · 2022-09-13T15:04:59Z

It´s a issue opened 23 June and it remains unsolved.
The related issues above dated from more than a year.

nk9 · 2022-11-08T12:20:46Z

I'm also running into this on Mac. The problem gets progressively more frequent until the Docker Desktop process (and thus vpnkit) is restarted. I've gone back to Docker Desktop 4.5.0 for the time being. Would really like to see this resolved so we can begin upgrading Docker again.

robertnisipeanu · 2022-11-08T14:27:16Z

This issue was fixed for me on MacOS after editing ~/Library/Group\ Containers/group.com.docker/settings.json and setting vpnKitMaxPortIdleTime from 300 to 0 (Docker Desktop restart required after). I have changed this over a week ago and till now I did not encounter the issue again.

mavci · 2022-11-08T20:51:05Z

I was having this issue on linux server (not Docker Desktop) and it was fixed by changing network mode to host. But I see this as workaround, because we expect it to work normally with vpnkit and bridge network mode as well. I am still waiting for some update and fix about this issue.

DirkvanWijk · 2022-12-12T09:03:43Z

I can also confirm that since Docker 4.6 I'm having the same issues.

nk9 · 2022-12-13T09:35:17Z

Sorry for the unsolicited tag, @djs55 and @avsm, but I was hoping to get some visibility on this. It's affecting lots of Docker Desktop users, who are sticking with 4.5 (February 2022) for now as a workaround. There are repro steps here and in the linked issue (docker/for-win#8861), but I haven't seen any acknowledgement that the VPNKit team is aware of the issue. Apologies if I missed it!

tristanbrown · 2023-02-17T19:08:23Z

This is still an issue. Worsening timeouts in Docker containers after they've been running for a few days. This bug is affecting countless developers across all fields, and should be prioritized.

tutcugil · 2023-04-10T16:53:39Z

We have the same issue, is there any update on this? It is very critical bug and still isn't resolved

djs55 · 2023-04-18T13:43:37Z

I've got an experimental developer build which might help. If you'd like to try it, it's here:

tutcugil · 2023-04-18T15:26:34Z

I've got an experimental developer build which might help. If you'd like to try it, it's here:

Intel Mac

Apple Silicon Mac

Windows

hello @djs55, thank you, i will try and inform you

Best Regards

tutcugil · 2023-04-24T08:28:07Z

I've got an experimental developer build which might help. If you'd like to try it, it's here:

Intel Mac

Apple Silicon Mac

Windows

hello @djs55, thank you, i will try and inform you

Best Regards

@djs55, our networking problem seems to resolve so far, we are still observing,
Will this experimental build release soon on docker?

OlegHudyma · 2023-05-03T21:22:02Z

@djs55 when can we expect for release version of this build?

I see that latest build version is 4.19 does not have this fixed

MrCSharp22 · 2023-06-18T09:49:38Z

This remains to be an issue on my end. Running docker in WSL2 or old-school using Hyper-V on Windows 11. Version 4.19 didn't resolve the problem. Currently on version 4.20.1 (Build 110738) and the frequency of outbound connection timeouts (including DNS queries to various DNS servers) seems to have increased.

Any updates to this will be very appreciated. I am having to restart the docker instance every couple of hours and this is very inconvenient.

Edit:

Adding some more info on fixes I tried so far:

Setting vpnKitMaxPortIdleTime to 0 from 300 didn't fix it.
Disabling vpnKit through settings.json didn't help.
Increased the number of ephemeral TCP/UDP ports that Windows can open for outbound connection to ~32k didn't help (by default, Windows allows for 16k).
Running docker on latest WSL2 or Hyper-V has the same issue.

Spenhouet mentioned this issue Nov 8, 2022

Docker network connection time outs to host over time docker/for-win#8861

Open

2 tasks

dougli mentioned this issue Nov 24, 2022

Connections aren't being closed openai/openai-python#140

Closed

DTM450 mentioned this issue Apr 25, 2023

Bug: [healthcheck] unhealthy: cannot dial: dial tcp4: lookup cloudflare.com: i/o timeout qdm12/gluetun#1164

Closed

scandey mentioned this issue May 7, 2023

UDP from Docker/Podman machine on Mac/Windows times out after 90 seconds OpenC3/cosmos#654

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random outbound connection timeouts based on server load #587

Random outbound connection timeouts based on server load #587

mavci commented Jun 23, 2022

rossinineto commented Jun 25, 2022

Spenhouet commented Jun 28, 2022 •

edited

jdeitrick80 commented Sep 5, 2022 •

edited

rossinineto commented Sep 13, 2022

nk9 commented Nov 8, 2022 •

edited

robertnisipeanu commented Nov 8, 2022

mavci commented Nov 8, 2022

DirkvanWijk commented Dec 12, 2022

nk9 commented Dec 13, 2022 •

edited

tristanbrown commented Feb 17, 2023

tutcugil commented Apr 10, 2023

djs55 commented Apr 18, 2023

tutcugil commented Apr 18, 2023

tutcugil commented Apr 24, 2023

OlegHudyma commented May 3, 2023

MrCSharp22 commented Jun 18, 2023 •

edited

Random outbound connection timeouts based on server load #587

Random outbound connection timeouts based on server load #587

Comments

mavci commented Jun 23, 2022

rossinineto commented Jun 25, 2022

Spenhouet commented Jun 28, 2022 • edited

jdeitrick80 commented Sep 5, 2022 • edited

rossinineto commented Sep 13, 2022

nk9 commented Nov 8, 2022 • edited

robertnisipeanu commented Nov 8, 2022

mavci commented Nov 8, 2022

DirkvanWijk commented Dec 12, 2022

nk9 commented Dec 13, 2022 • edited

tristanbrown commented Feb 17, 2023

tutcugil commented Apr 10, 2023

djs55 commented Apr 18, 2023

tutcugil commented Apr 18, 2023

tutcugil commented Apr 24, 2023

OlegHudyma commented May 3, 2023

MrCSharp22 commented Jun 18, 2023 • edited

Spenhouet commented Jun 28, 2022 •

edited

jdeitrick80 commented Sep 5, 2022 •

edited

nk9 commented Nov 8, 2022 •

edited

nk9 commented Dec 13, 2022 •

edited

MrCSharp22 commented Jun 18, 2023 •

edited