Docker: pull rate limit reached.

[mpranj]

Docker has recently implemented a pull rate limit for anonymous and free users. The limits are 100 (anon) and 200 (free) container image pull requests per six hours.

Builds are beginning to fail due to this limit and we will need to implement a fix or workaround.

docker build -t hub.libelektra.org/build-elektra-alpine:202012-0e6d95bb97e68999c969280c59562b159b8a0ecbee2a5aba451fe640081032de --pull --build-arg JENKINS_GROUPID=47110 --build-arg JENKINS_USERID=47110 --build-arg PARALLEL=12 --build-arg BASE_IMG=hub.libelektra.org/build-elektra-web-base:master_299 -f ./scripts/docker/alpine/3.12/Dockerfile ./scripts/docker/alpine/3.12
Sending build context to Docker daemon  6.144kB

Step 1/7 : FROM alpine:3.12.1
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
script returned exit code 1

[markus2330]

Our build server should actually only pull from our private docker registry, never from docker.org.

Is the problem maybe only a setting which we did not change on hub.libelektra.org? Or are there images which are not mirrored at hub.libelektra.org?

@robaerd can you please take a look? It is urgent, as it effects our builds.

[mpranj]

Or are there images which are not mirrored at hub.libelektra.org?

It seems that it always checks for updated base images which are not on our hub...

[robaerd]

It seems that it always checks for updated base images which are not on our hub...

This is part of the monthly rebuild of the docker images, since the month is part of the image id.

The docker images are currently cached again, so no rebuild of the docker images should happen and therefore the error should not reoccur at least this month.

I'm still not sure how we could exceed the 100 pull limit with our ~14 docker images.

[markus2330]

Thank you for looking into it. 💖 Yes, it looks a bit weird: if it is only 14 docker images and we pull only monthly we should be way below any limit?

[kodebach]

It seems that it always checks for updated base images which are not on our hub...

Is hub.libelektra.org configured like this: https://docs.docker.com/registry/recipes/mirror ? If so, my understanding is that the check whether the image is up-to-date should only count towards the quota, if it really needs to pull a new image.

The easiest way around the quota would be to create a Docker Hub Account for the CI. There is an Open Source Program, so probably we would be eligible for an unlimited account.

[markus2330]

I could do the application if it helps. But first we should find out what actually the problem is.

[kodebach]

I don't know how Docker Hub tracks the rate limit. I assume it is based on IP, otherwise it would be too easy to reset locally. In that case, is our build server the only thing that would appear to Docker Hub via this IP?

[markus2330]

Yes, the build server has a dedicated IP, even several ones, and the CI is the only part that uses docker.

[kodebach]

if it is only 14 docker images and we pull only monthly we should be way below any limit?

It seems that the Jenkins Pipeline runs a job (I think for the website) that tries to pull from Docker Hub all the time: https://build.libelektra.org/blue/organizations/jenkins/libelektra/detail/PR-3589/5/pipeline/696

AFAIK this happens because of the use of build --pull.

We should probably just use build (without --pull) by default and run it with --pull weekly or monthly.

[markus2330]

Thank you for finding out! 💖

[robaerd]

Thank you for finding the cause of this problem!

Alternatively to removing --pull, we could also create a base image for the webui base without elektra installed yet (only with the dependencies and gtests installed). This base image would be then - like the other ones - built monthly and the webui base image would extend from this base image and only pull from our private docker registry (and would thereby not affect the pull limit)

[markus2330]

webui base without elektra installed yet

I like this idea! Regardless of the Docker pull limits this would be an improvement!

[kodebach]

webui base without elektra installed yet

Yes that would also be an option. The image in question is already the base image for the actual webui and elektrad images. So we could just move the copying and building of Elektra into the other Dockerfiles. Or maybe there is a solution with multi-stage builds? Not sure, if intermediate stages can be pushed to/pulled from registries.

[robaerd]

Yesterday I was testing the shared library on jenkins where only the pull-stage was executed. No image building, only pulling from our private docker registry at hub.libelektra.org and I still got the Docker Rate Limit error. I looked a bit deeper and managed to find the cause of our problem.
It’s watchtower, a container running that updates our images at specified intervals. This issue should be fixed in their latest release. I will update this image and set the poll intervall to a higher value.
The logs of the watchtower container also confirm my assumption.

time="2020-11-16T22:22:58Z" level=info msg="Unable to update container /frontend_repo_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:22:59Z" level=info msg="Unable to update container /frontend_registry_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:00Z" level=info msg="Unable to update container /frontend_letsencrypt-nginx-proxy-companion_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:01Z" level=info msg="Unable to update container /frontend_nginx-proxy_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:02Z" level=info msg="Unable to update container /frontend_watchtower_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:04Z" level=info msg="Unable to update container /frontend_libelektra-webui_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:28Z" level=info msg="Unable to update container /frontend_repo_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:29Z" level=info msg="Unable to update container /frontend_registry_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."
time="2020-11-16T22:23:30Z" level=info msg="Unable to update container /frontend_letsencrypt-nginx-proxy-companion_1, err='Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit'. Proceeding to next."

[markus2330]

Thank you so much for finding out 💖

[markus2330]

@robaerd can we close this or is anything else to do?

[robaerd]

All docker images used in the artifact stage (webui, website, package tests) still pull from docker.org instead of our private registry. I think this should probably be in a separate issue since we would never exceed the docker pull-limit with this. But since the watchtower image update, this issue should be fixed and IMHO can be closed.

[markus2330]

Probably nothing else needs to be done. If we do not hit the limits, it is imho okay to pull from docker.org.