Hash sum mismatch #23203

herrbischoff · 2016-06-02T11:52:58Z

Steps to reproduce the issue:

Running apt-get update

Describe the results you received:
Running apt-get update on Debian Stretch just now results in

Err:2 https://apt.dockerproject.org/repo debian-stretch/main amd64 Packages
  Hash Sum mismatch

as well as

E: Failed to fetch https://apt.dockerproject.org/repo/dists/debian-stretch/main/binary-amd64/Packages.bz2  Hash Sum mismatch

I have cleaned the apt caches and tried again with the same result. Also, I'm not using a proxy.

Describe the results you expected:
No error.

The text was updated successfully, but these errors were encountered:

herrbischoff · 2016-06-02T12:00:18Z

Appears to be related to #23202.

mads79 · 2016-06-02T12:08:17Z

Same problem on Debian Trusty

W: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

nielsvandermolen · 2016-06-02T12:10:55Z

Similar problem on Travis CI with Ubuntu package. It was working an hour ago.

https://travis-ci.org/goalgorilla/drupal_social/builds/134719276

W: There is no public key available for the following key IDs:
1397BC53640DB551
W: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

zjeraar · 2016-06-02T12:13:40Z

Same on Debian Jessie:
W: Failed to fetch https://apt.dockerproject.org/repo/dists/debian-jessie/main/binary-amd64/Packages Hash Sum mismatch

zjeraar · 2016-06-02T12:15:54Z

Can easily be repoduced in a contatiner also:

FROM debian:8.4

RUN \
  apt-get update && \
  apt-get install -yq apt-transport-https ca-certificates && \
  apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
  echo "deb https://apt.dockerproject.org/repo debian-jessie main" > /etc/apt/sources.list.d/docker.list && \
  apt-get update

Vanuan · 2016-06-02T12:22:14Z

What about apt-get clean? Does it help?

ivanfeob · 2016-06-02T12:23:27Z

@Vanuan No, already tried.

GordonTheTurtle · 2016-06-02T12:23:40Z

USER POLL

The best way to get notified of updates is to use the Subscribe button on this page.

Please don't use "+1" or "I have this too" comments on issues. We automatically
collect those comments to keep the thread short.

The people listed below have upvoted this issue by leaving a +1 comment:

@ViGo5190

mcapuccini · 2016-06-02T12:23:41Z

Same here!

Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  Hash Sum mismatch

gillesdemey · 2016-06-02T12:25:23Z

Things we've tried so far:

Re-add the GPG key
curl -fsSL https://get.docker.com/gpg | sudo apt-key add -

Blow the lists cache
sudo rm -rf /var/lib/apt/lists/*

Apt clean
apt-clean

None of them have resolved the issue

jraut · 2016-06-02T12:27:40Z

Tried to install via apt. Checksum mismatch with the following file:

https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages

Tried the following procedures which did not help:

apt-clean

sudo rm -rf /var/lib/apt/lists/*

Vanuan · 2016-06-02T12:27:48Z

Maybe this would help apt-get -o Debug::pkgAcquire::Auth=true update figure out the issue.

thoastbrot · 2016-06-02T12:30:14Z

Release contains:
MD5Sum:
49df2d605bb5914873fd826f7e7e8c6f 4917 Packages.bz2

InRelease contains:
b013253c327e2bc4be87825f02936344 4915 main/binary-amd64/Packages.bz2

the latter has been updated today, Date: Thu, 02 Jun 2016 11:06:54 UTC
while Release is from yesterday.

gillesdemey · 2016-06-02T12:32:33Z

Running apt-get -o Debug::pkgAcquire::Auth=true update on Ubuntu 14.04 yields

[Waiting for headers]201 URI Done: bzip2:/var/lib/apt/lists/partial/apt.dockerproject.org_repo_dists_ubuntu-trusty_main_binary-amd64_Packages
RecivedHash: SHA512:d6ca1f74e876031161d1abd6cf9ad0b45f60b19876468cfcf9cacd4956dfd13be43147227a8daa5536f1455bb75b353b178942bc1843d11f0188d00117483912
ExpectedHash: SHA512:d07a3f2c42a9b213e3f03f2f11c08154512baa9fbbaed19f3601865634b82cfdde0e65151a24e523017f29ecfd08a1dfc0af2c2117b025c46d683160892b0de6


https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages: 
Computed Hash: SHA512:d6ca1f74e876031161d1abd6cf9ad0b45f60b19876468cfcf9cacd4956dfd13be43147227a8daa5536f1455bb75b353b178942bc1843d11f0188d00117483912  
Expected Hash: SHA512:d07a3f2c42a9b213e3f03f2f11c08154512baa9fbbaed19f3601865634b82cfdde0e65151a24e523017f29ecfd08a1dfc0af2c2117b025c46d683160892b0de6

herrbischoff · 2016-06-02T12:32:41Z

Relevant output of apt-get -o Debug::pkgAcquire::Auth=true update:

Got Codename: debian-stretch
Expecting Dist:
Transformed Dist:
Signature verification succeeded: /var/lib/apt/lists/partial/apt.dockerproject.org_repo_dists_debian-stretch_InRelease
Get:2 https://apt.dockerproject.org/repo debian-stretch/main amd64 Packages [4,941 B]
0% [Connecting to ftp.de.debian.org] [Connecting to security.debian.org] [Connecting to mirror.netcologne.de] [Connecting to packages.dotdeb.org] [Connecting to www.deb-multimedia.org] [Connecting to ftp-stud.hs-esslingen.de] [Connecting201 URI Done: https://apt.dockerproject.org/repo/dists/debian-stretch/main/binary-amd64/Packages.bz2
ReceivedHash:
    - SHA512:14844ddc767052951fb68eabc19a1935fb930c798d64fd86ace0dcce3aad2af887fc091ad90897a52f341f65dadac5f0dc31a35f9c70b5bcc582314187a336cf
    - SHA256:0cee3ef5330e133cc6dfbf3d34f118806ce685a1ded4210c5c4f7ef7b43e9867
    - SHA1:bcf84731c3d9fe4355ce73b3cd756decbf9b67cb
    - MD5Sum:c99614887831f4d020e682c8222fe49b
    - Checksum-FileSize:4933
ExpectedHash:
    - Checksum-FileSize:4941
    - SHA512:5de62937921a32be2e9cf14f65e6adda3499fd648f37ab5ccc9547a03d211be66c3a5cd15f272e5a3f0abc53fec3903f646410337917e4201bf2a7ed5ac8581d
    - SHA256:ebc0ec8921482f40bdcf1fa9a7f39b7bd198d81a769643723201c109b3b617ea
    - SHA1:a61818ebafdccbccdfdeee5e550b9241b8c32722
    - MD5Sum:9cd9390adc1849ba5923a70d92af1927

nielsvandermolen · 2016-06-02T12:35:05Z

https://travis-ci.org/goalgorilla/drupal_social/builds/134730044

Get:11 https://apt.dockerproject.org ubuntu-trusty/main amd64 Packages
201 URI Done: https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages.bz2
RecivedHash: SHA512:36e068ae0288732c51bd971ee74b6d27c8707f4d11840afcca617884de82e8c533c5259d8d97bb297966424bc58ac219879f4f5d12c4abe073799bb658f4bd87
ExpectedHash: SHA512:d07a3f2c42a9b213e3f03f2f11c08154512baa9fbbaed19f3601865634b82cfdde0e65151a24e523017f29ecfd08a1dfc0af2c2117b025c46d683160892b0de6

bmoorthamers · 2016-06-02T12:39:41Z

I get on Ubuntu Wily 15.10

E: Unable to locate package docker-engine

I got the same before on Ubuntu Xenial 16.04. Is docker even added to the Xenial repo yet?

ThomasSteinbach · 2016-06-02T12:40:50Z

Relevant output of apt-get -o Debug::pkgAcquire::Auth=true update :

Got Codename: ubuntu-xenial
Expecting Dist: 
Transformed Dist: 
Signature verification succeeded: /var/lib/apt/lists/partial/apt.dockerproject.org_repo_dists_ubuntu-xenial_InRelease
Holen:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [1.712 B]
Ign:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Holen:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [1.430 B]
Ign:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Holen:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [4.815 B]
100% [12 Packages 4.815 B/4.815 B 100%]201 URI Done: https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/Packages
ReceivedHash:
    - SHA512:c7883bb7a1d0b5162431576408644a85003be4601724b6f2db275cd4b603a61f8dcd924e80158c40413942519c8a528f7940ffbe5370daa4b0a0d867afe3163d
    - SHA256:de12840d76e571cb6f42e63ac570c59d5332d772fb295b6919d12214052bfa6b
    - SHA1:9f9c05d3b7d8ca13e9e03c4f0f12757816f02301
    - MD5Sum:65e1f5c451c230a091118b468c31bae7
    - Checksum-FileSize:4815
ExpectedHash:
    - Checksum-FileSize:4815
    - SHA512:2becf6c2b9aae5b6823ea6d9f12988e22905a87a9a03fed844a761698eee614899d7b039e081e0b330539e716918b75e87a96c287a5efbe9fc3e847d44657798
    - SHA256:f4ae20e2259740699fba3a79dd7fb557c472d172b578798071274f7ba4c400f3
    - SHA1:8f34563e8170c5698dc7ba04dd3cf4c8a93100cf
    - MD5Sum:31d143b7a15a8a38bc92a7559c995078

thoastbrot · 2016-06-02T12:41:44Z

can we agree upon the fact that the hashsums are incorrect/repo needs administrative action?

qzio · 2016-06-02T12:42:27Z

I worked around this by downloading the new package manually and installing using dpkg

curl -OL https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.11.2-0~trusty_amd64.deb
dpkg -i docker-engine*.deb

OJFord · 2016-06-02T12:48:10Z

Unfortunately dpkg installation doesn't seem to play well on Travis.

herrbischoff · 2016-06-02T12:49:47Z

Also, this is what happens to me when manually installing on Debian Stretch, using https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.11.2-0~stretch_amd64.deb:

$ sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2016-06-02 14:46:59 CEST; 58s ago
     Docs: https://docs.docker.com
 Main PID: 31269 (code=exited, status=1/FAILURE)

Jun 02 14:46:58 penny systemd[1]: Starting Docker Application Container Engine...
Jun 02 14:46:58 penny docker[31269]: time="2016-06-02T14:46:58.553905409+02:00" level=info msg="New containerd process, pid: 31293\n"
Jun 02 14:46:59 penny docker[31269]: time="2016-06-02T14:46:59.659258835+02:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Jun 02 14:46:59 penny docker[31269]: time="2016-06-02T14:46:59.659395935+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Jun 02 14:46:59 penny systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 14:46:59 penny docker[31269]: time="2016-06-02T14:46:59+02:00" level=info msg="stopping containerd after receiving terminated"
Jun 02 14:46:59 penny systemd[1]: Failed to start Docker Application Container Engine.
Jun 02 14:46:59 penny systemd[1]: docker.service: Unit entered failed state.
Jun 02 14:46:59 penny systemd[1]: docker.service: Failed with result 'exit-code'.

Update: As I somehow expected, this was an unrelated problem. I fixed it by running rm -rf /var/lib/docker/aufs after finding this. So the manual install works for me for the time being.

thaJeztah · 2016-06-02T12:54:30Z

ping @mlaventure @tiborvass PTAL!

akersj · 2016-06-02T13:02:40Z

ETA?

rvetere · 2016-06-02T13:03:55Z

yes we need an ETA too, it's pretty urgent - our complete travis build chain is dead now -.-

simos · 2016-06-02T13:03:59Z

Here are the relevant files for xenial,
https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/

InRelease        02-Jun-2016 11:06  2.6K
Packages          02-Jun-2016  2:38  4.8K
Packages.bz2  02-Jun-2016  2:38  1.7K
Packages.gz    02-Jun-2016  2:38  1.4K
Release            02-Jun-2016  3:43  1.7K
Release.gpg    02-Jun-2016  3:43  801

We can see that these files have been regenerated earlier today.
The checksums (hashes) for these files should match what is inside the signed InRelease file of checksums.

In the InRelease file (https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/InRelease), it says that this file was generated on Date: Thu, 02 Jun 2016 03:43:32 UTC . However, the timestamp as shown by the Web server is 02-Jun-2016 11:06.

Among the several causes for Hash Sum Mismatch, this one is about some weird update of InRelease with wrong checksums. In addition, InRelease lists the Release as being at 0 bytes long.

bmoorthamers · 2016-06-02T13:06:34Z

@simos So this should work on Xenial now? I thought docker still did not work on Xenial and we had to go back to Wily. (then again, I am an Ubuntu user since today so what do I know)

simos · 2016-06-02T13:10:58Z

@bmoorthamers You can manually check which repositories have mismatched hashes. See my post above. At least trusty, wily and xenial are currently (probably since earlier in the morning) affected.

cpuguy83 · 2016-06-02T15:22:14Z

@vadviktor Does not work at Docker nor maintain the project.

stefano-m · 2016-06-02T15:22:42Z

@qzio see http://www.docker.com/docker-support-services

quater · 2016-06-02T15:23:24Z

It appears to be working for Ubuntu Xenial now.

okket · 2016-06-02T15:23:37Z

FYI, this issue is on HN https://news.ycombinator.com/item?id=11822562

oryband · 2016-06-02T15:25:10Z

for everybody raging over this downtime: here's a cute deer picture to calm down and pass the time in the meanwhile:

mech422 · 2016-06-02T15:25:45Z

Trusty appears to be back up

shykes · 2016-06-02T15:25:49Z

Hi everyone. I work at Docker.

First, my apologies for the outage. I consider our package infrastructure as critical infrastructure, both for the free and commercial versions of Docker. It's true that we offer better support for the commercial version (it's one if its features), but that should not apply to fundamental things like being able to download your packages.

The team is working on the issue and will continue to give updates here. We are taking this seriously.

Some of you pointed out that the response time and use of communication channels seem inadequate, for example the @dockerststus bot has not mentioned the issue when it was detected. I share the opinion but I don't know the full story yet; the post-mortem will tell us for sure what went wrong. At the moment the team is focusing on fixing the issue and I don't want to distract them from that.

Once the post-mortem identifies what went wrong, we will take appropriate corrective action. I suspect part of it will be better coordination between core engineers and infrastructure engineers (2 distinct groups within Docker).

Thanks and sorry again for the inconvenience.

mech422 · 2016-06-02T15:28:03Z

heh - got the catalog but the package is missing - guess I'll have another cofee :-)
@shykes Thanks for the update - lousy way to start your morning ...
Hope the day gets better from here

oryband · 2016-06-02T15:30:02Z

I'm sad that my deer picture got less +1s than the official answer.

Rocklviv · 2016-06-02T15:31:31Z

I have already installed docker with https://get.docker.com | sh without any error.
Look's like the guys from Docker fixed the issue,

thaJeztah · 2016-06-02T15:31:32Z

We localized the cause of the issue, and if should be resolved now, please try again.

It may be needed to clear the apt-cache;

apt-get clean && apt-get update

snario · 2016-06-02T15:32:29Z

Thanks for the fix @thaJeztah

LouisKottmann · 2016-06-02T15:33:45Z

Well, that was quick for an unexpected issue, thanks.

thaJeztah · 2016-06-02T15:33:55Z

@snario you're welcome; can't take credit for the fix, but happy to see it's been sorted out 😅

rogaha · 2016-06-02T15:38:48Z

👍

jamescarr · 2016-06-02T15:42:26Z

Sadly with this on the top of hacker news there's going to be billions of comments. Big thanks for the quick fix @thaJeztah.

I wonder if we should lock this thread before they show up.

simos · 2016-06-02T15:49:09Z

Up until now there have been workarounds (either grab the .deb and install with dpkg, temporarily switch to the testing repository, etc). These are not permanent solutions.

A fix means that the source of this problem is resolved and we can mark this issue as Solved.

As posted earlier, you can use a script to get an audio notification as soon as the main docker repositories are fixed,
https://gist.github.com/simos/7ee8258ec17101e44bbfa93606694ede
Other than that, there is not much to do.

thaJeztah · 2016-06-02T15:53:55Z

@simos see my earlier comment; #23203 (comment) the issue should be resolved

denibertovic · 2016-06-02T16:15:27Z

@thaJeztah I verified that the issue has been resolved. Tested on Ubuntu 15.10. Thanks to all the other Docker folks that helped resolve this issue quickly.

icecrime · 2016-06-02T16:18:08Z

Thank you all for the reports: we're very sorry for this. We're looking into the details and the timeline of events that lead to this, and we'll make sure it doesn't happen again.

I'm closing the issue, but of course feel free to let me know if you see any remaining quirks.

Niceplace · 2016-06-02T16:41:48Z

Ubuntu 14.04 Here, issue solved !

jalawrence · 2016-06-02T18:15:51Z

Probably shouldn't be surprised, but it is shocking how many people risk their infrastructure with hard dependencies on external repos. I don't even do that with my home systems.

And then complain about Docker having a single point of failure?

gggeek · 2016-06-02T18:36:34Z

@jalawrence Docker is the tip of the iceberg...
Did you hear about the recent problems with node.js and one dev pulling out one single package?
I am pretty sure that most php developers using Composer - the defacto package manager for that platform - also do not store complete copies of all their site's dependencies, and the fact that there has been no mishaps so far is more luck than anything.
The problem is that everybody and their dog now depends on $world, and caching all the dependencies locally is a sisyphean task. Shall I cache all of debian, all of packagist, all of cpan, all of rubygems, all of npm within a reverse proxy at my own expenses?
And then: if github, bitbucket or travis are down, what will my developers be able to do anyway? Do I want back to the day when I had to host all of that?

herrbischoff changed the title ~~Has sum mismatch~~ Hash sum mismatch Jun 2, 2016

mcapuccini pushed a commit to mcapuccini/SparkNow that referenced this issue Jun 2, 2016

back to trusty repo, opened an issue on docker moby/moby#23203

5b3b6ab

tbfisher mentioned this issue Jun 2, 2016

skip provisioning during create? docker/machine#3473

Open

pascalgrimaud mentioned this issue Jun 2, 2016

Travis: use new system matrix cache by Travis jhipster/generator-jhipster#3678

Merged

icecrime closed this as completed Jun 2, 2016

yosifkit mentioned this issue Jun 2, 2016

Xenial docker image creates/observes bogus hash sum mismatches. tianon/docker-brew-ubuntu-core#51

Closed

BanzaiMan mentioned this issue Jun 2, 2016

Build error due to failed to fetch index? travis-ci/travis-ci#6125

Closed

yosifkit mentioned this issue Jun 2, 2016

Docker-machine AWS driver on us-east-1 Error creating machine: Error running provisioning: error installing docker: docker-library/official-images#1798

Closed

moby locked and limited conversation to collaborators Jun 2, 2016

Hash sum mismatch #23203

Hash sum mismatch #23203

Comments

herrbischoff commented Jun 2, 2016 • edited

herrbischoff commented Jun 2, 2016 • edited

mads79 commented Jun 2, 2016

nielsvandermolen commented Jun 2, 2016 • edited

zjeraar commented Jun 2, 2016

zjeraar commented Jun 2, 2016

Vanuan commented Jun 2, 2016

ivanfeob commented Jun 2, 2016

GordonTheTurtle commented Jun 2, 2016

mcapuccini commented Jun 2, 2016 • edited

gillesdemey commented Jun 2, 2016

jraut commented Jun 2, 2016 • edited

Vanuan commented Jun 2, 2016

thoastbrot commented Jun 2, 2016 • edited

gillesdemey commented Jun 2, 2016 • edited

herrbischoff commented Jun 2, 2016

nielsvandermolen commented Jun 2, 2016

bmoorthamers commented Jun 2, 2016 • edited

ThomasSteinbach commented Jun 2, 2016

thoastbrot commented Jun 2, 2016

qzio commented Jun 2, 2016 • edited

OJFord commented Jun 2, 2016

herrbischoff commented Jun 2, 2016 • edited

thaJeztah commented Jun 2, 2016

akersj commented Jun 2, 2016

rvetere commented Jun 2, 2016

simos commented Jun 2, 2016

bmoorthamers commented Jun 2, 2016

simos commented Jun 2, 2016 • edited

cpuguy83 commented Jun 2, 2016

stefano-m commented Jun 2, 2016

quater commented Jun 2, 2016

okket commented Jun 2, 2016

oryband commented Jun 2, 2016

mech422 commented Jun 2, 2016

shykes commented Jun 2, 2016

mech422 commented Jun 2, 2016

oryband commented Jun 2, 2016

Rocklviv commented Jun 2, 2016

thaJeztah commented Jun 2, 2016

snario commented Jun 2, 2016 • edited

LouisKottmann commented Jun 2, 2016

thaJeztah commented Jun 2, 2016

rogaha commented Jun 2, 2016

jamescarr commented Jun 2, 2016 • edited

simos commented Jun 2, 2016

thaJeztah commented Jun 2, 2016

denibertovic commented Jun 2, 2016

icecrime commented Jun 2, 2016 • edited

Niceplace commented Jun 2, 2016

jalawrence commented Jun 2, 2016

gggeek commented Jun 2, 2016

herrbischoff commented Jun 2, 2016 •

edited

herrbischoff commented Jun 2, 2016 •

edited

nielsvandermolen commented Jun 2, 2016 •

edited

mcapuccini commented Jun 2, 2016 •

edited

jraut commented Jun 2, 2016 •

edited

thoastbrot commented Jun 2, 2016 •

edited

gillesdemey commented Jun 2, 2016 •

edited

bmoorthamers commented Jun 2, 2016 •

edited

qzio commented Jun 2, 2016 •

edited

herrbischoff commented Jun 2, 2016 •

edited

simos commented Jun 2, 2016 •

edited

snario commented Jun 2, 2016 •

edited

jamescarr commented Jun 2, 2016 •

edited

icecrime commented Jun 2, 2016 •

edited