Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider doing the db upgrade in an init container #1885

Open
artntek opened this issue Apr 19, 2024 · 0 comments
Open

Consider doing the db upgrade in an init container #1885

artntek opened this issue Apr 19, 2024 · 0 comments
Milestone

Comments

@artntek
Copy link
Contributor

artntek commented Apr 19, 2024

Problem: initial DB upgrade makes metacat unresponsive, so k8s restarts the pod

We should consider doing the db upgrade in an init container, so that metacat won't be started until the upgrade is done

For the knb corpus, the initial DB upgrade from 2.19.0 -> 3.0.0 took about 3-1/2 hours on the dev cluster (metacat running on k8s-dev-node-3 (slower machine); postgres on k8s-dev-node-5 (faster machine)).

During this time, liveness and readiness probes timed out because metacat was completely unresponsive, so k8s restarted the pod.

The workaround was temporarily to set:

livenessProbe:
  enabled: false
readinessProbe:
  enabled: false

and then switch back after the upgrade was complete.

Note that log4j was set to DEBUG level for the first 3 hours. Not sure if this slowed things down significantly?

@artntek artntek modified the milestones: 3.1.0, 3.0.0 Apr 19, 2024
@artntek artntek assigned artntek and unassigned artntek Apr 19, 2024
@taojing2002 taojing2002 modified the milestones: 3.0.0, 3.1.0 Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants