Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a service for easy data updates #109

Open
ellenhp opened this issue Aug 23, 2022 · 0 comments
Open

Create a service for easy data updates #109

ellenhp opened this issue Aug 23, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@ellenhp
Copy link
Member

ellenhp commented Aug 23, 2022

This init container thing we do right now doesn't scale well to cover the planet. It's not reasonable to have one copy of each artifact per pod when the artifacts are tens or hundreds of gigabytes. My current plan is to create a longhorn RWX volume claim and share that between all of the pods. Each pod will mount the volume claim as read-only and serve data out of it. I'll have to write a new service to manage the data that's kept in the pvc and help do the following:

  1. download new data when it becomes available
  2. restart the deployments as necessary when they need to start serving new data
  3. determine when it's safe to delete old data by examining running pods
  4. actually delete the old data once it's safe

That service will need to be able to access the cluster's api (kubelet? still getting up to speed on terminology) for items 2 and 3.

@ellenhp ellenhp added the enhancement New feature or request label Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant