Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPG error: https://packages.sury.org/php bullseye InRelease: Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bullseye_InRelease into data and signature failed #2046

Open
3 tasks done
rfay opened this issue Oct 28, 2023 · 17 comments

Comments

@rfay
Copy link

rfay commented Oct 28, 2023

Frequently asked questions

  • I have read Frequently Asked Questions
  • I have looked at the list of the existing issues (including closed issues) and searched if my issue has been already reported
  • I have tried to resolve the issue myself and will describe what I did in clear and consise manner

Describe the bug

I know you've probably heard this too much, but in DDEV's automated tests I often see this, but it's quite intermittent. There can be 6 tests running in parallel on github workflows and two of them might hit it. I've seen it locally as well, and done things like changing networks to try to get around it.

[web 3/5] RUN apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -o Dpkg::Options::="--force-confold" --no-install-recommends --no-install-suggests tmux:
 4.056 W: GPG error: https://packages.sury.org/php bullseye InRelease: Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bullseye_InRelease into data and signature failed
4.058 E: The repository 'https://packages.sury.org/php bullseye InRelease' is not signed.

To Reproduce

I do not know how to reproduce it, as it's very intermittent, but it does happen every day.

Your understanding of what is happening

I suspect there must be one cache or one CDN that is invalid somehow.

I would love to know what "Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bullseye_InRelease into data and signature failed" means.

What steps did you take to resolve issue yourself before reporting it here

I have already done PRs that I hoped would resolve this in DDEV, using less things from deb.sury.org, etc. But since this happens on apt update, my hoped solutions were a failure.

Expected behavior

apt update should always work, and not fail to "split"

Distribution (please complete the following information):

  • OS: Debian
  • Architecture: both amd64 and arm64
  • Repository: packages.sury.org

Package(s) (please complete the following information):

The package is not relevant, as the problem happens on apt update

Additional context

  • I do think something is wrong somewhere networking-wise. It's not likely on the client side, as this happens in multiple build environments on multiple networks.
  • This only happens on the deb.sury.org repo, not on any other that is in the apt lists
  • /etc/apt/sources.list.d/php.list has what you would expect, deb [signed-by=/usr/share/keyrings/deb.sury.org-php.gpg] https://packages.sury.org/php/ bullseye main
@oerdnj
Copy link
Owner

oerdnj commented Oct 28, 2023

I think it might make sense for you to do a local mirror of the packages. There's rsync available at rsync.sury.org

@rfay
Copy link
Author

rfay commented Oct 28, 2023

None of this is local, it's all on various CI environments like GitHub Actions etc.

@rfay
Copy link
Author

rfay commented Oct 29, 2023

I should note that people all over the world can hit this when using DDEV, which may add a build step including apt-get update at runtime. So it's not just a testing problem, and not something that I know how to solve with a mirror or anything.

I have a PR going trying to gather more information, ddev/ddev#5473

But can this be something about DNS or b-cdn.net? Is there possibly one bad IP address returned for debsuryorg.b-cdn.net. ?

@deviantintegral
Copy link

I saw this error today. In this case, apt was running inside of GitHub actions. I agree it's intermittent, as other jobs passed as did a rebuild.

@rfay
Copy link
Author

rfay commented Feb 5, 2024

I was able to fix this in one particular situation after an extended bit of trouble by changing wireless networks (to phone hotspot). I wish I knew what the error message actually meant.

@rfay
Copy link
Author

rfay commented Feb 7, 2024

I had this problem consistently today on a coffee-shop wifi.

I looked at DNS server settings, and I was using 1.1.1.1 (Cloudflare).

I changed to use the coffee-shop's default DNS and it worked fine.

@oerdnj
Copy link
Owner

oerdnj commented Feb 7, 2024

I had this problem consistently today on a coffee-shop wifi.

I looked at DNS server settings, and I was using 1.1.1.1 (Cloudflare).

I changed to use the coffee-shop's default DNS and it worked fine.

Hmm, but you haven't recorded the DNS responses by any chance?

@rfay
Copy link
Author

rfay commented Feb 7, 2024

Sorry, that would have been a good test.

I switched back to 1.1.1.1 and now I don't seem to be able to recreate the problem.

The response below is working.

$ nslookup -type=ns packages.sury.org
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.

Authoritative answers can be found from:
debsuryorg.b-cdn.net
	origin = ns1.bunnydns.com
	mail addr = hostmaster.bunnydns.com
	serial = 2020612118
	refresh = 7200
	retry = 900
	expire = 1209600
	minimum = 86400

rfay@rfay-tag1-m1:~/workspace/ddev/containers/ddev-webserver$ nslookup packages.sury.org
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.
Name:	debsuryorg.b-cdn.net
Address: 84.17.63.178

@penyaskito
Copy link

I'm seeing this error today consistenly. In this case, apt is running inside of GitHub actions.

@deviantintegral
Copy link

@penyaskito on GitHub hosted runners, our self hosted ones at Linode, or both?

@rfay
Copy link
Author

rfay commented Feb 20, 2024

This issue is about the "Splitting up" message.

It's not related to the cert expiration message that @penyaskito was getting due to actually having an expired key. That problem is covered in

The answer for @penyaskito was to update DDEV to v1.22.7. There is also a workaround in the issue for DDEV v1.22.6.

@rfay
Copy link
Author

rfay commented May 31, 2024

I'm having this error right now, happening on both amd64 and arm64, on two networks:

GPG error: https://packages.sury.org/php bookworm InRelease: Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bookworm_InRelease into data and signature failed

$ nslookup packages.sury.org 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.
Name:	debsuryorg.b-cdn.net
Address: 84.17.63.178

$ nslookup packages.sury.org 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.
Name:	debsuryorg.b-cdn.net
Address: 84.17.63.178

On another internet-based system where apt update is working I see

# nslookup packages.sury.org 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.
Name:	debsuryorg.b-cdn.net
Address: 169.150.221.147

All I can figure here is that my ISP is spoofing 1.1.1.1 and 8.8.8.8

If I add a line to /etc/hosts with
169.150.221.147 packages.sury.org

everything works fine.

@rfay
Copy link
Author

rfay commented Jun 1, 2024

I switched to hotspot, completely different network, and I still get the same failure, different IP address

$ nslookup packages.sury.org 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.
Name:	debsuryorg.b-cdn.net
Address: 84.17.63.178

rfay@d10-web:/var/www/html$ nslookup packages.sury.org 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
packages.sury.org	canonical name = debsuryorg.b-cdn.net.
Name:	debsuryorg.b-cdn.net
Address: 212.102.40.114

@rfay
Copy link
Author

rfay commented Jun 1, 2024

But maybe it's bunnydns? It either isn't serving, or isn't willing to talk to me:

$ nslookup packages.sury.org ns1.bunnydns.com
Server:		ns1.bunnydns.com
Address:	157.53.226.1#53

*** Can't find packages.sury.org: No answer

@rfay
Copy link
Author

rfay commented Jun 1, 2024

I guess maybe this is something about BunnyCDN caching @oerdnj ?

On the affected systems I'm seeing a 304 "no need to re-transmit" with a curl:

% curl -I https://packages.sury.org/php/dists/bookworm/InRelease
HTTP/2 304
date: Sat, 01 Jun 2024 20:22:59 GMT
server: BunnyCDN-DEN1-919
cdn-pullzone: 717719
cdn-uid: a7a277f7-2828-404b-9c94-f3b9b03c0434
cdn-requestcountrycode: US
cache-control: public, proxy-revalidate, max-age=7200
expires: Sat, 01 Jun 2024 22:06:18 GMT
last-modified: Tue, 14 May 2024 20:00:45 GMT
cache-tag: metadata,dists
cdn-proxyver: 1.04
cdn-requestpullsuccess: True
cdn-requestpullcode: 304
cdn-cachedat: 06/01/2024 20:06:18
cdn-edgestorageid: 919
cdn-status: 304
cdn-requestid: 0e84a891e6127db41f669675e232145b
cdn-cache: HIT

On unaffected systems I see a 200:

$ curl -I https://packages.sury.org/php/dists/bookworm/InRelease
HTTP/2 200
date: Sat, 01 Jun 2024 20:25:28 GMT
content-type: application/octet-stream
content-length: 7542
server: BunnyCDN-TX1-881
cdn-pullzone: 717719
cdn-uid: a7a277f7-2828-404b-9c94-f3b9b03c0434
cdn-requestcountrycode: US
cache-control: public, proxy-revalidate, max-age=7200
expires: Sat, 01 Jun 2024 22:06:42 GMT
last-modified: Tue, 14 May 2024 20:00:45 GMT
cache-tag: metadata,dists
cdn-proxyver: 1.04
cdn-requestpullsuccess: True
cdn-requestpullcode: 206
cdn-cachedat: 06/01/2024 20:06:42
cdn-edgestorageid: 950
cdn-status: 200
cdn-requestid: da088787700c024d7c9b552262700e31
cdn-cache: HIT
accept-ranges: bytes

@rfay
Copy link
Author

rfay commented Jun 1, 2024

As I think we'd expect here, on a working system we get this from a curl, fully PGP-signed:

curl -s https://packages.sury.org/php/dists/bookworm/InRelease
curl  -s  https://packages.sury.org/php/dists/bookworm/InRelease
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Origin: deb.sury.org
Suite: bookworm
Codename: bookworm
Date: Tue, 14 May 2024 20:00:45 UTC
Architectures: amd64 i386 arm64 armhf
Components: main
Signed-By: 15058500A0235D97F5D10063B188E2B695BD4743
MD5Sum:
 fbe4ad12865414b255e966ebddeb8b0f 1817232 main/binary-amd64/Packages
 5d27727440be7770248ba373dc68a7a6 402365 main/binary-amd64/Packages.gz
 e83ddd0b3723ccc618f621b825e07151 229188 main/binary-amd64/Packages.xz
 b164231371f38edfd6ccbcb1f74f5a89 75 main/binary-amd64/Release
 285136698027c1849e1afff39e9e1391 1772641 main/binary-i386/Packages
 1db3ab23c023611241b662204918a0e1 391545 main/binary-i386/Packages.gz
 fc1cd9229bfbb6a30ec0bda9d9bce6a0 223916 main/binary-i386/Packages.xz
 97351b0ba73773198270c10448a65cf8 74 main/binary-i386/Release
 f98171de29795ddc813a4151b76c0042 1761166 main/binary-arm64/Packages
 5561899527871280f54db4a0e0ce55ee 387778 main/binary-arm64/Packages.gz
 de2f01bb423708e45a1a878f0d1ddf9d 221428 main/binary-arm64/Packages.xz
 da08a4cc3f54b325f1ef609a9dd8d910 75 main/binary-arm64/Release
 be3c6e1c759b43beee09d445dfd9be2a 1750141 main/binary-armhf/Packages
 13bafc7b696680b86bc854fe678174c7 385288 main/binary-armhf/Packages.gz
 2495137ad39cb70209cbc7a548d7765e 220668 main/binary-armhf/Packages.xz
 ca3cd33d520115b831cdf660983895f6 75 main/binary-armhf/Release
 0f5e562821a460523bf0f03b2615763a 275942 main/source/Sources
 c3dd3d6473cd71b75beeb77ee778a7d5 55175 main/source/Sources.gz
 148520af633d8895c2968d47f0e87cc1 45816 main/source/Sources.xz
 5944aeb204de3bdf10909ca7ed897616 76 main/source/Release
 f679dcde3c636fbf50f685225fe132df 822432 main/Contents-amd64
 15d52fc5c01b72a0ff6ee1d3c84df6f4 100943 main/Contents-amd64.gz
 3ccef546f2d95ca7caa55ec9e13347ad 763619 main/Contents-i386
 b6d4ea5c5cecf82b2e12dbd3ebfcc746 93571 main/Contents-i386.gz
 8f593c588e9e826134a954388159e399 788166 main/Contents-arm64
 d625e2376a62a98e9f358634139c0f4d 96763 main/Contents-arm64.gz
 ac120f3984b40394a576f977258f79eb 757370 main/Contents-armhf
 0f65cb1b005d191a6400d7212ecfdab6 92676 main/Contents-armhf.gz
SHA1:
 1ccb54098918cfbdf54a66e0ca04df746dd70cde 1817232 main/binary-amd64/Packages
 bf80004733e224e09c7b6568d2914650616e3f7b 402365 main/binary-amd64/Packages.gz
 37895949047ebbb3be658c2daab949e6a258dac6 229188 main/binary-amd64/Packages.xz
 6cc4b93c817b160e8709c55463d404b052adb432 75 main/binary-amd64/Release
 a87c2bb3a92d3e821edc581aa8c80cc92ebf7e08 1772641 main/binary-i386/Packages
 5b98e4d716e80115602a28fc52c4e6fc2364338c 391545 main/binary-i386/Packages.gz
 5c7e5bb14c906ed1221f359dd1d4c9e6798c15fa 223916 main/binary-i386/Packages.xz
 4aea3ef2a5a5d704829ad1cf8c336eafe04db689 74 main/binary-i386/Release
 ed03226083ad91c9608bf53878533474fb064079 1761166 main/binary-arm64/Packages
 8cc5d7dfb419b6729ef1ca0a8ada6faf5b227477 387778 main/binary-arm64/Packages.gz
 4d931ef85c27937a9a48d232959058f8bc9a4246 221428 main/binary-arm64/Packages.xz
 84e556b06d3eab8a63d3c65509fb056f5d6b6609 75 main/binary-arm64/Release
 b19c9b999e12675faf844abf81809a1d1812b1fd 1750141 main/binary-armhf/Packages
 11f11f01fbd6891fe78fbe0a9902252d232fbdd7 385288 main/binary-armhf/Packages.gz
 18b0c5b9c5dcb3cfe514f6676a738f092823c9bf 220668 main/binary-armhf/Packages.xz
 60ea84a18d8379dbb68105ef5cd2973cab3f07c7 75 main/binary-armhf/Release
 c553d56f8e6247f0ccb463a356f916bde4f2be1f 275942 main/source/Sources
 950904ad44d39220a72e2f753034e5b7da210640 55175 main/source/Sources.gz
 92d1e1fd0b90aede2d0ffd21bf248d1a37856efc 45816 main/source/Sources.xz
 4ad2a32f9353d48a027f89bb6dfb559f3789edd3 76 main/source/Release
 1c6bbd3f4194994380bfe1c4a7e9e723aee65f0a 822432 main/Contents-amd64
 3af05b330e2a81e8242789b9e5a6b9d9874ffcd5 100943 main/Contents-amd64.gz
 c63c10a662d98c051e5ca130a79ec3cc6295f7e8 763619 main/Contents-i386
 a13ce421d57b5e214a9c4dd38ea5c761854e3cc3 93571 main/Contents-i386.gz
 af76507b6d88b04aef4c5dd6dec8bc9cae19647b 788166 main/Contents-arm64
 28e1863d418c7f24a379f7583aac3e22e2f39f7a 96763 main/Contents-arm64.gz
 008dfa23325d289788eb23956eba025789ede534 757370 main/Contents-armhf
 be995505d4d59c5a3d684ea4f439b2f96f1ee372 92676 main/Contents-armhf.gz
SHA256:
 fe8c2db11b2cca11102fdf6eb312071a143ab99915988e10d66914747241b052 1817232 main/binary-amd64/Packages
 2f9772ab4846b77c8e37cc8d04866d577a882206b86b7a60811b21e051cfd807 402365 main/binary-amd64/Packages.gz
 df8e8be847cadfb7d764d582a6864f04933c4c9e13428ac1b7e64563812db179 229188 main/binary-amd64/Packages.xz
 d561db745afd6a5f0b405eaf73bcec647b93ba60143f30a34e1b18ffb0a22014 75 main/binary-amd64/Release
 7904358d0612ec6d4e8a529a9a71d6a2da3872d790903467f05978faa8963172 1772641 main/binary-i386/Packages
 a298f2e00d3e763e22d4e860752a45c3f83a8bf8e7cde767ad0e102305f51e14 391545 main/binary-i386/Packages.gz
 87e7f4807e0d2700a0461c0286f1084d87b97443068a63f00474d01184293af4 223916 main/binary-i386/Packages.xz
 01e4b67a49e26c78439f3f4c0b71d610c7a7519eafb2dbe4ba405f9162a076ca 74 main/binary-i386/Release
 8604e488817f3a7d1184c5e3f87f9a8170452522ede3ff05c8818b95ab825e98 1761166 main/binary-arm64/Packages
 4cc9be38940ce637e58e9f8270ab30a5a0a3277b313cde971438582a3b37c822 387778 main/binary-arm64/Packages.gz
 b25c08d88b9e2dbea3c259b065d0b3547154424a890e0d2f134b1c503d1deede 221428 main/binary-arm64/Packages.xz
 ea9f3cfd5cf0e8c856230d53f73164b60e2be50b7b2ac0099e476faef2254d04 75 main/binary-arm64/Release
 4ee5705bcd0790e8b0f44826cd3a7f8960aebb624c3ec03378c616d3abe6e23b 1750141 main/binary-armhf/Packages
 e4625cb03139a2b60bf3a703bbbc4c1695f060cd88a5010f8f85cfab624b4558 385288 main/binary-armhf/Packages.gz
 8b6f802f47f82adec8754931c020215f16eae9a570117e68112ed65b05bd5629 220668 main/binary-armhf/Packages.xz
 64294113ee9a74e2ca2e7540ab47176e748060e6de0934f46263c46ecb719c94 75 main/binary-armhf/Release
 3b935b7a2e8ec0ce8804860403013350f810908ba71ece05ff2468577bf7aa24 275942 main/source/Sources
 c66429d5a2ff61b38ccc1c1cc4d07292b3d250ad29ff8cc1670c1595b385eea9 55175 main/source/Sources.gz
 81b7df711df330ac9b44306d47436697b3d7363c7d3e31de4e762d0d4bd997c6 45816 main/source/Sources.xz
 2c398967fb30860157907e6170cb6a90eed7c9391f33ea14b94898746f50a94e 76 main/source/Release
 630998b100b8b352fae39a05c492f01cfc0afc5f3274c21e66918e0c1d3de9f0 822432 main/Contents-amd64
 0993aba68c23993f730d927467ba5f07436f40abe662fd73733eb25f1d2558fc 100943 main/Contents-amd64.gz
 638c2d11900c0abbf680c9d0a163c6424538b30ca6ad4260f6fa3f98744f37b2 763619 main/Contents-i386
 a8c7e414341093a1d36a238fb3f730e095334566aff435ea0c4eb539f9dd45f2 93571 main/Contents-i386.gz
 280b9a12a1dcd0bfde9e5e88fd998c8fbf049ca09150b9868410112e758d54db 788166 main/Contents-arm64
 5f15b5cb8ea0534fb3555d21786c986e79d217cab39037f7d43eb4d307c48cd7 96763 main/Contents-arm64.gz
 3f34cad47c9590419f08f69960c5c43623112bee355abdcab96572d1ed6877b7 757370 main/Contents-armhf
 c5dde5edc05d6299795c9ae07aae441e1b21d359c5d1d78c09d1c2508ec09c8b 92676 main/Contents-armhf.gz
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEEFQWFAKAjXZf10QBjsYjitpW9R0MFAmZDwu0ACgkQsYjitpW9
R0MfkQv/ZLRojfUhrVj5eoLh8yCUFpr14Q81jr2XBBTEsJ3rhOLLoLOmId8T/a8K
Wq5oPK9zjXq97zJn2FlakVU/W0Li7J4X33gNhjah48CojBDcHb21XQZZGpqMX05Y
7Il2g6HUq7SpBRTc/CLMJ4OOx30kSLO0/7iZG06PHtAb3VFoHaOlcnlrgyXQbwGb
SrXe3xomFTVbLDAtWtqakPfx7mDrVR5kwYQHwUh7o0hYjNcAfgow0BRCPMVgpnQd
O9X0YoN0hwX8uDk6Se4pOoU92fhUfnsWBq9bvm0k2rZDhaxzW29dARkGOO3PdbUb
Pzknx72iww9djLKdV4IyMzw2gmPm4Qf6j8eAKbdohb4cWL5BjF2xmJj5/xAUjmx2
G8yHgd8NJl0Wsxe7en4Gqn+TPpy9E00QsiJotgpjj4tPSzrPkL12x6CTpsyNsSzV
QNVrhEio4FwUjZHjXaJx1mkOMyAb8PLd5oeYjS7cWjeAL6mc1LxcRzpcRC5Rq3QG
y0PO0zce
=WZ8F
-----END PGP SIGNATURE-----

On a broken system as expected, because of the 304, we get absolutely nothing back:

% curl  -s  https://packages.sury.org/php/dists/bookworm/InRelease
%

@oerdnj
Copy link
Owner

oerdnj commented Jun 2, 2024

I’ll open a ticket with BunnyCDN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants