Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers #22635

Closed
almoehi opened this issue May 10, 2016 · 74 comments

Comments

@almoehi
Copy link
Contributor

almoehi commented May 10, 2016

Output of docker version:

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 53
 Running: 0
 Paused: 0
 Stopped: 53
Images: 331
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 437
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 3.13.0-86-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66 GiB
Name: 3kolor
ID: BAOT:3C5I:WCTI:RXDF:5W55:BJTA:MRDP:NSAW:4H4R:ILDC:BG3Z:C65X
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 12
 Goroutines: 35
 System Time: 2016-05-10T19:54:31.370342285+02:00
 EventsListeners: 0
Username: almoehi
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 14.04.4 LTS
Linux 3.13.0-86-generic #130-Ubuntu SMP Mon Apr 18 18:27:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the issue:

  1. docker login

Describe the results you received:
It seems like docker daemon cannot access the internet, even though internet access is working on the box. I cant pull images and I cant even log-in to the registry.
I've re-installed the whole docker packages and rebooted the machine several times - no success.

docker login:

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (******): 
Password: 
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

manually running a request with CURL works without problems:

 curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

running docker pull ubuntu just hangs forever without producing any debug output in the daemon logfile and after couple of minutes dies with following message:

Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"

Describe the results you expected:
Docker daemon should be able to access the internet. Docker should successfully login to registry and pull images.

Additional information you deem important (e.g. issue happens only occasionally):

debug logfile

time="2016-05-10T19:56:01.669665420+02:00" level=debug msg="Calling GET /v1.23/info" 
time="2016-05-10T19:56:05.459596278+02:00" level=debug msg="Calling POST /v1.23/auth" 
time="2016-05-10T19:56:05.459709902+02:00" level=debug msg="form data: {\"email\":\"*****\",\"password\":\"*****\",\"serveraddress\":\"https://index.docker.io/v1/\",\"username\":\"******\"}" 
time="2016-05-10T19:56:05.459899278+02:00" level=debug msg="attempting v2 login to registry endpoint https://registry-1.docker.io/v2/" 
time="2016-05-10T19:56:06.429620835+02:00" level=debug msg="Increasing token expiration to: 60 seconds" 
time="2016-05-10T19:56:20.928767529+02:00" level=error msg="Handler for POST /v1.23/auth returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" 

debug logfile when running docker pull:

time="2016-05-10T20:03:52.257721929+02:00" level=error msg="Error trying v2 registry: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\\n\"" 
time="2016-05-10T20:03:52.257784072+02:00" level=error msg="Attempting next endpoint for pull after error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\\n\"" 
time="2016-05-10T20:03:52.257835365+02:00" level=error msg="Handler for POST /v1.23/images/create returned error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\\n\""
@ouyi
Copy link

ouyi commented Jun 2, 2016

Exactly the same issue here.

@jsainzvela
Copy link

jsainzvela commented Jun 3, 2016

+1 on 1.11.2 under Ubuntu 14.04.4 LTS

@jsainzvela
Copy link

Hi, I think we have hit the #15949 issue

Lowering my server's MTU to 1500 has worked around the problem.

Cheers

@almoehi
Copy link
Contributor Author

almoehi commented Jun 8, 2016

I already hat MTU size of 1500 and still not working.

Apparently, enabling MTU probing in the kernel resolved the issue. To enable it just run:

 sudo sysctl -w net.ipv4.tcp_mtu_probing=1

To make this permanently, add the following line to /etc/sysctl.conf:

net.ipv4.tcp_mtu_probing=1

https://thesimplecomputer.info/pages/adventures-in-linux-tcp-tuning-page2

@almoehi almoehi closed this as completed Jun 8, 2016
@justincormack
Copy link
Contributor

Glad you found a solution.

@luxas
Copy link

luxas commented Aug 28, 2016

I'm also getting this issue:

Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Not continuing with pull after error: Network timed out while trying to connect to https://index.docker.io/v1/repositories/armhf/busybox/images. You may want to check your internet connection or if you are behind a proxy.

docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.19-1-ARCH
Operating System: Arch Linux ARM
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 922.2 MiB
Name: pi3arch
ID: 7MTA:UZIW:SVAS:H3U7:YBHE:FWXV:KWOS:4QAQ:HDLG:EWIV:DOAP:TWFR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

I've set the mtu to 1500 and net.ipv4.tcp_mtu_probing=1 with no success :(
Any ideas?

@MathiasRenner
Copy link

MathiasRenner commented Oct 13, 2016

Same issue here as reported by @luxas.

@LipingMao
Copy link

Same problem here, mtu already be 1500 and net.ipv4.tcp_mtu_probing=1 with no success.
@luxas @MathiasRenner

@luxas
Copy link

luxas commented Nov 11, 2016

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

@Nhoui
Copy link

Nhoui commented Dec 29, 2016

Thank you luxas
It works great!!!

@Devendrabhat
Copy link

@luxas I worked for my linux system, but not getting what to do on windows. Please help if possible.

@diptiburli
Copy link

facing the same issue here changed the resolv.conf , modified mtu size .. no use. docker is still able to contact the registry.

@diptiburli
Copy link

docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
Am seeing the above error

@chenrui333
Copy link

chenrui333 commented Jan 26, 2017

@diptiburli I had the similar issue, what I did to resolve that:

  1. update docker and docker-machine to the latest version, 1.13.0
  2. brew link --overwrite docker and docker-machine
  3. docker-machine restart {VM}

Although I still have error with curl https://registry-1.docker.io/v2/, but the docker pull works for me:

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Let me know if this works for you.

@aasaanhai
Copy link

For Windows 10 Home, Putting 8.8.8.8 as the first dns resolves it.

@kirstin
Copy link

kirstin commented Jan 30, 2017

@aasaanhai also worked for me.
Simply set the default DNS to 8.8.8.8 on the "vEthernet (DockerNAT)" network adapter.

@strainel
Copy link

strainel commented Feb 6, 2017

thanks but where simply set the default DNS to 8.8.8.8 ?
on debian for exemple... thanks again

@JohnVonNeumann
Copy link

@strainel Should be on your network connections area, you're running linux so it should be similar to my ubuntu, use the GUI to go into your network settings, select the connection you're on, and then put the google dns settings in there. I don't have practical experience with debian but a google search should do the trick.

@strainel
Copy link

thanks @JohnVonNeumann i solved my issue.
the message "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" was caused by network connection with a proxy. with https://docs.docker.com/engine/admin/systemd/ (§ http proxy), it is ok.
i am surprised docker use systemd on debian... and not environment variables.
thanks again

@cooligc
Copy link

cooligc commented Mar 2, 2017

+1 in RHEL 7

[root@minion2 yum.repos.d]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.03.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 2.781 GiB
Name: minion2
ID: Q6KC:VGM5:DE7P:I2TW:H2SQ:A3CI:GFRM:IPLZ:MKA5:SHJI:6CLP:4XSU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
[root@minion2 yum.repos.d]#

@pronojitsaha
Copy link

Setting the DNS server to fixed (8.8.8.8) from automatic in docker settings solved the problem for me.
dns

@ravargs
Copy link

ravargs commented Mar 10, 2017

Hello,

I am still facing same issue. The server is behind the corporate proxy but woks for all other applications.

export http_proxy=http://user:password@proxyIP:8080/
export https_proxy=http://user:password@proxyIP:8080/
docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: xxxxxxx
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

@ravargs
Copy link

ravargs commented Mar 10, 2017

I got it solved by adding the proxy details in the docker config file /etc/default/docker.

@ratchakr
Copy link

Same problem faced running docker-machine on windows. Restarting docker-machine worked for me suggested by chenrui333. My docker version is 1.13.1

docker-machine restart

docker login as well as docker pull is working now.

@elrondwong
Copy link

Same problem here, mtu already be 1500 and net.ipv4.tcp_mtu_probing=1 with no success.dns=8.8.8.8 also no use

@GArgiz
Copy link

GArgiz commented Apr 21, 2017

I am getting the same issue...
I have tried change the mtu (1500,1400,1300) and nothing happens. There is not a proxy issue neither.
Can someone help ?
Thanks

Unable to find image 'mysql:latest' locally docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Tue Mar 28 00:40:02 2017
OS/Arch: darwin/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Fri Mar 24 00:00:50 2017
OS/Arch: linux/amd64
Experimental: false

@matchilling
Copy link

matchilling commented Apr 26, 2017

I've experienced the above described problem this morning on my Ubuntu 16.04 machine (4.4.0-75-generic) as well.

$ docker version
Client:
 Version:      17.04.0-ce
 API version:  1.28
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.04.0-ce
 API version:  1.28 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64
 Experimental: false

After login with sudo rights (sudo docker login -u username -p password), which apparently worked, I've deleted the docker config (sudo rm ~/.docker/config.json) and restarted the docker service (sudo /etc/init.d/docker restart). For some magical reason I was now able to login again without super user rights.

Hope that helps.

@taostrive
Copy link

root@xxxx:# cat /etc/issue
Ubuntu 16.04 LTS \n \l
root@xxxx:
# docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:14:09 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:14:09 2017
OS/Arch: linux/amd64
Experimental: false
root@xxxx:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.

In my case, setting the environment variable http_proxy directly in shell and then run systemctl restart docker doesn't work at all, but we need to follow the instructions related to proxy server settings in https://docs.docker.com/engine/admin/systemd/, then it works!

@shajay12
Copy link

Hi Everyone,
Request every one to give any solution to me.
I have implemented everything that is written in this thread, but still the i am getting this error
"Client.Timeout exceeded while awaiting headers"
I am using Ubuntu 17.10.

Regards,
Aj

@supernicky
Copy link

Hi, in my case setting a fixed DNSmade things worse, the main course was the companys proxy (as always :) ). Try to change the proxy to a working one or working without one.

@shajay12
Copy link

shajay12 commented Jun 13, 2018 via email

@inevity
Copy link

inevity commented Jul 2, 2018

@alldev0825 thanks. For old docker 1.13, can use /etc/sysconfig/docker to set proxy ,but for 17.06 docker-ce, it does not work,need setup proxy in the systemd dir as the link you supplied.

@shajay12
Copy link

shajay12 commented Jul 2, 2018 via email

@kapiljh
Copy link

kapiljh commented Sep 24, 2018

I already hat MTU size of 1500 and still not working.

Apparently, enabling MTU probing in the kernel resolved the issue. To enable it just run:

 sudo sysctl -w net.ipv4.tcp_mtu_probing=1

To make this permanently, add the following line to /etc/sysctl.conf:

net.ipv4.tcp_mtu_probing=1

https://thesimplecomputer.info/pages/adventures-in-linux-tcp-tuning-page2

Thank you.. It fixed the issue 💯

@sureshgururajan
Copy link

sureshgururajan commented Dec 1, 2018

I added below configuration on /etc/resolv.conf as suggested above and it works

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

this one worked for me! thank you so much :)

@anirudhkoutha
Copy link

docker login to private registry in our environment is failing intermittently.
We are having same issue with private registry.
Configuring Google's DNS would work for docker.io but for internal registries how can we extend the login timeout value, esp with slow AD Authorizations.

We have configured our docker NGINX with proxy_ignore_client_abort on; to ignore client timeouts to verify if there is a server kick outs. But the client looks hard coded with 15s timeout irrespective of the server behaviour. Any way we can change/extend this timeout?

@jasonyilkk
Copy link

one thing to point out: using docker login from Docker: check /etc/resolv.conf which is used by daemon to resolve the registry. Not the /etc/hosts.

@Light-Gao
Copy link

Same Issue.
With docker Server Version: 17.03.1-ce on CentOS Linux release 7.4.1708 (Core).
Got mtu as 1500, net.ipv4.tcp_mtu_probing=1 and modified /etc/resolv.conf file, no success.

Please anyone light me. Thanks a lot.

@fxnn
Copy link

fxnn commented Mar 29, 2019

On my Win10, changing the DNS does only work because the Docker daemon restarts to apply the change. In other words: restarting the daemon helps, no matter what DNS server is configured.

@kiahmed
Copy link

kiahmed commented Mar 29, 2019

what if i dont have a dns when the registry is local in the lan. then manually adding reg server to /etc/hosts file worked for me

@sksam2147
Copy link

sksam2147 commented Nov 2, 2019

I just simply changed the nameserver /etc/resolv.conf
nameserver 192.168.1.1
nameserver 8.8.8.8
nameserver 8.8.4.4
Thank god, It worked for me. Thank you!

@ghost
Copy link

ghost commented Nov 23, 2019

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

This worked for me thanks

@Sumitkaroo
Copy link

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 192.168.0.1

after editing to

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10
This worked for me.

@tsaibabu4u
Copy link

for me below steps worked

Create a systemd drop-in directory for the docker service:

sudo mkdir -p /etc/systemd/system/docker.service.d
Create a file named /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80"
If you are behind an HTTPS proxy server, set the HTTPS_PROXY environment variable:

[Service]
Environment="HTTPS_PROXY=https://proxy.example.com:443"
Multiple environment variables can be set; to set both a non-HTTPS and a HTTPs proxy;

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80"
Environment="HTTPS_PROXY=https://proxy.example.com:443"

then do

sudo systemctl stop docker
sudo systemctl daemon-reload
sudo systemctl start docker

@dodjango
Copy link

I get exactly this message with linux containers on windows 10 after updating to docker desktop 3.0.0 and enabling wsl integration. Windows containers works.

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (******): 
Password: 
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Playing with all the solutions above did not help (dns, mtu, proxy, reconfigure network / firewall)

Uninstalling and reinstalling the current version of docker desktop fixed the issues.

@0x1za
Copy link

0x1za commented Feb 5, 2021

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

Thank you @Sumitkaroo

@owos
Copy link

owos commented Jun 28, 2022

Hi, I have your second error, how did you solve that?

@vieerD
Copy link

vieerD commented Jul 6, 2022

it did worked thanks

@imjavierpalma
Copy link

imjavierpalma commented Jul 12, 2022

I had the same issue. I use Arch Linux. I went to $HOME/.docker and I deleted the config.json file. Then I type sudo systemctl restart docker and sudo systemctl enable docker. After that I type docker login and put my username and docker token and BOOM!

I hope it helps!

@avinashvagh
Copy link

@alldev0825 thanks. For old docker 1.13, can use /etc/sysconfig/docker to set proxy ,but for 17.06 docker-ce, it does not work,need setup proxy in the systemd dir as the link you supplied.

can you help me to fix my docker problem ? I've the same issue.

@shyamgupta196
Copy link

@ghostplant where can I find /etc/sysconfig/docker this file, I am also facing the same error

@1250529731
Copy link

I can't use docker to loaddown nginx,who can give me some issue
I can't use docker to loaddown nginx,who can give me some issue

@1250529731
Copy link

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

thank you bro

@1250529731
Copy link

I use his idea and solve my problem,let's take a look
docekr2
docker1
docker3

@alexisba
Copy link

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

is still valid december 2023

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests