Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plndr-cp-lock failure with kube-vip pod when backing up K3s sqlite database #572

Open
yebo29 opened this issue Feb 29, 2024 · 0 comments
Open

Comments

@yebo29
Copy link

yebo29 commented Feb 29, 2024

I'm trying to use Litestream to backup the db of my 3-node k3s cluster running on Raspberry Pi 4s. It's a single primary node cluster with two worker nodes.
I've written Ansible code to deploy the litestream systemd service to the primary node and backup to Backblaze. I can confirm that the service runs, connects to my bucket and starts replicating to it.
However, I notice that certain pods start restarting when it runs. I've tried different --sync-intervals with no success.
Specifically, the most detrimental of these is the kube-vip pod that dies with the following errors:

E0228 20:52:15.433056       1 leaderelection.go:369] Failed to update lock: Put "https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plndr-cp-lock?timeout=10s": context deadline exceeded
I0228 20:52:15.434181       1 leaderelection.go:285] failed to renew lease kube-system/plndr-cp-lock: timed out waiting for the condition
error: http2: client connection lost

During the pod restart process, I lose access to the VIP and API. I do not see any errors in the logs for litestream when I run journalctl -xf -u litestream, just replication logs. However, upon writing this I realize that by default, log level is set to INFO. Perhaps I can temporarily change that and observe. But in the meantime, I want to see if anyone else in the community has seen this, and if there are any solutions I can try.

I'm running the latest tag of kube-vip, v0.7.1

Below is my current (redacted) config:

access-key-id: [redacted]
secret-access-key: [redacted]

dbs:
  - path: /var/lib/rancher/k3s/server/db/state.db
    replicas:
      - type: s3
        bucket: mybucket
        endpoint: s3.us-xxxx-xxx.backblazeb2.com
        path: litestream
        force-path-style: true
        sync-interval: 30s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant