Kellnr calculates wrong checksum [CRITICAL]

[alexthe2]

Since last week (there was no particular event), kellnr started calculating the wrong checksums for the crates pushed to it.

This results in cargo not being able to verify checksums.

failed to verify the checksum of `v12-data v1.17.1 (registry `kellnr`)

As an example crate:

sqlite> select * from crate_index where name="v12-data" AND vers="1.17.1";
id|name|vers|deps|cksum|features|yanked|links|v|crate_fk
35|v12-data|1.17.1|...|90392cdc01079ee500391dd0fd409059158b83b170567361d2e3552b76450f6c|{}|0||1|5

The checksum here is not the checksum that would occur from running sha256sum, when we did a UPDATE crate_index ... with the result of sha256, that particular crate started correctly working (with cargo).

We're happy to help with resolving this issue asap, as it's a criticial bug for us, but would need some guidance as to where this issue could be originating from.

We're running the latest kellnr 5.2.1, with rust 1.77.2 on the buildserver (Jenkins) that pushes to kellnr, and locally 1.75+

[secana]

Hi @alexthe2! Thanks for reporting the issue. Does this happen only with one specific crate or under specific circumstances? I'll try to reproduce the issue.

[alexthe2]

It happens with two crates for us (from what I can see right now), called v12-data and v12-terra_converters, v12-terra_converters is depndent on data

[secana]

Can you provide logs from Kellnr in the "trace" mode, when the issue occurs? I try to replicate the wrongly computed hash, but so far without any success.

[alexthe2]

should I set KELLNR_LOG__LEVEL or KELLNR_LOG__LEVEL_WEB_SERVER?

[secana]

KELLNR_LOG__LEVEL should be enough.

[alexthe2]

it's not producing any logs 😢 , I verfied that the level is really set to trace (updated helm chart, and deleted pod to force restart)

[secana]

I still try to debug the issue but have no idea why the sha256 is computed wrong. Can you try to disable the cache, so I know that it is not a caching issue?

KELLNR_REGISTRY_CACHE_SIZE=0

[asymmetry]

Hi @secana, we are experiencing the same issue. I could provide one more data point.

Compare the sha256sum of a corrupted crate in the db and the actual value:
In db:

Actual:

And I am already running with KELLNR_REGISTRY__CACHE_SIZE=0

I am running Kellnr 5.2.2 with the released docker image.

Thanks!

[asymmetry]

If I delete the corrupted version from the web ui as admin user, restart the docker container, and then publish the same crate again, it could fix the problem.

[alexthe2]

Fun story for us, if you hand fix the cksum in SQLite, it might solve the problem for the next version. The reason I say might is it solved it for one crate but not for all of them

…

On Sat, May 11, 2024, 12:18 AM Chao Gu ***@***.***> wrote: If I delete the corrupted version from the web ui as admin user, restart the docker container, and then publish the same crate again, it could fix the problem. — Reply to this email directly, view it on GitHub <#311 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIBZJB247NNNPAJFMTXG2BTZBVBSNAVCNFSM6AAAAABHDRHCLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGMZTMNJZGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[secana]

Thanks for the input. So far, the issue seems to be the computation of the sha256, as the crate itself seems to be fine on disk. I released a debug version of kellnr with much more debug output for the specific issue. Would you be so kind and try to run it and provide the logs here?

Kellnr version: 5.2.3-debug-311
Helm chart version: 3.2.3-debug-311

All logs are in the level debug and prefixed with #311 to be easily identifiable. For the debug version, I added an additional crate to compute the sha256 and be able to compare it with the current implementation. Hopefully the debug output shows us the right direction to finally find and fix the issue.

[asymmetry]

Thanks @secana! I have deployed this test version with logs enabled. I tried to publish a test crate 10 times (with slight modification each time) and the logs looks good to me. My team will continue to use this version and I will show the logs here if the issue happens again.

Thanks for the help!

[secana]

New version of Kellnr 5.3.2 is out with improved SHA256 computation. This error should be fixed with that version.

[alexthe2]

Also a finishing update from our side, we got the debug version running two days ago, no issues yet, we'll now switch to the new 5.3.2

[asymmetry]

Hi @secana, my team have the debug version running for almost 10 days and we captured this issue again. Here is the log:

It seems that the error is happenned when reading the crate saved to disk back into memory for sha256 calculation, it only reads 4096 bytes.
However it seems you have already switched to use the in-memory data to calcualte checksum so I think this is not going to happen again. We will switch to the new 5.3.2.
Thanks a lot for the help!