Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

service gets stuck and doesn't update anymore after some time of uninterrupted operation #106

Open
paolobarbolini opened this issue Mar 12, 2021 · 15 comments · May be fixed by #227
Open

service gets stuck and doesn't update anymore after some time of uninterrupted operation #106

paolobarbolini opened this issue Mar 12, 2021 · 15 comments · May be fixed by #227
Labels
bug Something isn't working

Comments

@paolobarbolini
Copy link
Member

After a few weeks of deps.rs running without new deploys or restarts it looks like the updater for popular crates and popular repos gets stuck and doesn't update anymore. This has happened multiple times now.

This is easily noticeable when serde releases a new version, since the homepage keeps pointing to the older version, causing the badge to report 1 outdated dependency (serde imports an exact version of serde-derive).

deps.rs

remote

local instance

local

@paolobarbolini paolobarbolini added the bug Something isn't working label Mar 12, 2021
@paolobarbolini paolobarbolini changed the title Homepage gets stuck and don't update anymore after a few weeks of uninterrupted operation Homepage gets stuck and doesn't update anymore after a few weeks of uninterrupted operation Mar 12, 2021
@Feliix42
Copy link
Member

Any ideas on how this can happen? Maybe a caching issue?

@robjtede
Copy link
Member

Anything useful logged? Does restarting the deployment help?

@paolobarbolini
Copy link
Member Author

Does restarting the deployment help?

It does for a few weeks and then it happens again.

@Feliix42
Copy link
Member

Feliix42 commented Mar 26, 2021

Currently, everything is up to date (presumably due to the merge of #107). Let's see, how long it lasts.

@paolobarbolini
Copy link
Member Author

It happened again

@Feliix42
Copy link
Member

Feliix42 commented Aug 9, 2022

So I guess it makes sense to assume that both #156 and #145 are somehow related to this?

As @kid pointed out, the logs show only a few error messages regarding timeouts, but nothing beyond that. Maybe it makes sense to try and locate the problem methodically:

  • How does the deployment machine look after deps has been running a while?
    • Is there a slow but steady increase in memory usage for instance?
    • Are we accidentally keeping any connections open?
  • Can we get a more detailed error logging set up? Or can we get a full log that spans the time before and after the error start occuring? Sadly the instance restart also means we lost the logs, I think.
  • Can we try to reproduce the problem locally on our machines by spamming it for a while with requests?
  • @cecton pointed out that we could be rate-limited. Can we check that? Since the log snippet posted by @kid shows both Gitlab and Github timeout-ing, I would guess it's not on the receiving end of our requests at least.

@kid
Copy link
Member

kid commented Aug 9, 2022

image
definitely looks like a memory leak

@cecton
Copy link
Member

cecton commented Aug 9, 2022

If we were hitting the memory limit the process would have been killed with signal 9 and I would expect the thing to restart automatically and to be working again.

@robjtede
Copy link
Member

robjtede commented Aug 9, 2022

Couldn't it just be the cache getting bigger over time, as we'd expect?

@crjeder
Copy link

crjeder commented Sep 13, 2022

For me it currently looks like this:
image
And the badge for my crate does time out, too.

@crjeder
Copy link

crjeder commented Sep 13, 2022

For me it currently looks like this: image And the badge for my crate does time out, too.

works again!

@mumbleskates
Copy link

there it goes again

@robjtede robjtede changed the title Homepage gets stuck and doesn't update anymore after a few weeks of uninterrupted operation Homepage gets stuck and doesn't update anymore after some time of uninterrupted operation May 26, 2024
@robjtede robjtede changed the title Homepage gets stuck and doesn't update anymore after some time of uninterrupted operation service gets stuck and doesn't update anymore after some time of uninterrupted operation May 26, 2024
@paolobarbolini
Copy link
Member Author

I've tried running deps.rs locally with the following patches:

deps.rs

diff --git a/Cargo.lock b/Cargo.lock
index 9721623..4aeff6a 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -245,8 +245,6 @@ dependencies = [
 [[package]]
 name = "crates-index"
 version = "2.10.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8467f424ab6d70e5daf384289fa68247787effc13901dd0ca46bc9b4a62f1474"
 dependencies = [
  "gix",
  "hex",
diff --git a/Cargo.toml b/Cargo.toml
index 746daaf..d8f3589 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -16,7 +16,7 @@ badge = { path = "./libs/badge" }
 
 anyhow = "1"
 cadence = "1"
-crates-index = { version = "2", default-features = false, features = ["git"] }
+crates-index = { path = "../rust-crates-index", default-features = false, features = ["git"] }
 derive_more = "0.99"
 font-awesome-as-a-crate = "0.3"
 futures-util = { version = "0.3", default-features = false, features = ["std"] }

rust-crates-index

diff --git a/src/error.rs b/src/error.rs
index 7ae1d27..5f06ee2 100644
--- a/src/error.rs
+++ b/src/error.rs
@@ -7,7 +7,7 @@ pub use toml::de::Error as TomlDeError;
 #[derive(Debug, thiserror::Error)]
 #[allow(missing_docs)]
 pub enum Error {
-    #[error("\"gix\" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/`")]
+    #[error("\"gix\" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` {0:?}")]
     #[cfg(feature = "git")]
     Git(#[from] GixError),
     #[error("{0}")]

The logs were the following:

May 26 16:20:53.511 INFO Server running on port 8080
May 26 16:41:13.861 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:42:33.871 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:46:33.881 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:51:33.890 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:57:33.904 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:58:33.921 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 17:08:54.265 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:09:14.315 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:09:34.308 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:09:54.263 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:10:14.286 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:10:34.286 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:10:54.309 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:11:14.261 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:11:34.265 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:12:13.930 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 17:12:14.285 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:12:34.282 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:12:54.274 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:13:14.306 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:13:34.292 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:13:54.299 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:14:14.338 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:14:34.272 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:14:54.340 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))

@robjtede
Copy link
Member

robjtede commented May 26, 2024

ObtainRefDuringIteration(
  NotFound {
    oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb),
    name: "refs/heads/master"
  }
)

This reads to me like it's having issues with the repo collapses.

I ran current master locally and saw the same kind of errors as you, Paolo. Following the advice of the error message (deleting ~/.cargo/registry/index/github.com-...) I'm yet to see it resurface.

Possible solutions and workarounds:

  • we run rm -rf ~/.cargo/registry/index/github* as part of the restart cronjob
  • we refactor the service to use the sparse registry

@robjtede robjtede linked a pull request May 26, 2024 that will close this issue
@robjtede robjtede linked a pull request May 26, 2024 that will close this issue
@robjtede
Copy link
Member

refactoring to sparse index wasn't too much work: #227

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants