Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store blobs outside database #585

Open
matthew-white opened this issue Jan 11, 2024 · 2 comments · May be fixed by getodk/central-backend#1071
Open

Store blobs outside database #585

matthew-white opened this issue Jan 11, 2024 · 2 comments · May be fixed by getodk/central-backend#1071
Assignees
@alxndrsn
Labels
backend Requires a change to the API server enhancement New feature or behavior needs testing Needs manual testing

Comments

@matthew-white
Copy link
Member

matthew-white commented Jan 11, 2024 •
edited by lognaturel

User stories

As a Central systems administrator with CLI access to the server
I want all blobs on my server to be stored in S3-compatible storage
So that I can back it up separately from my database

As a Central systems administrator with CLI access to the server
I want all existing blobs in my server database to be migrated to S3
So that I can reduce the size of my existing database

As a Central systems administrator with CLI access to the server
I want blobs to be purged from S3-compatible storage when the last resource that references them is purged
So that I can keep my storage usage down

As a Central systems administrator with CLI access to the server
I want to a way to see counts of blobs uploaded and blobs pending
So that I can follow up on any failures that happened

As a Central systems administrator with CLI access to the server
I want a way to force uploading db blobs to S3
So that I can control when that migration happens (e.g. to match a scheduled maintenance window)

As an ODK data collector or data analyst
I don’t want to know anything about where blobs are stored
So that I can focus on the tasks I’m an expert on

As an ODK developer
I want to get analytics reports on how many blobs are stored in the db vs storage
So that I can iterate on this feature area according to user needs

Expectations

  • If a blob fails to transfer, it will seamlessly continue to be served from the database and will be retried within a week
  • Trying to upload the same blob multiple times has no effect (e.g. if I restore a db backup that still had blobs that already had been migrated)

Out of scope

These stories could add value but are currently considered out of scope.

As a Central administrator with CLI access to the server
I want to migrate from storing media in S3-compatible storage back to storing it in the database
So that I can reduce the complexity of my deployment

As a Central administrator with CLI access to the server
I want Central to manage backups of my media in S3-compatible storage
So that I can have a unified backup and restore experience

As an ODK project manager
I want to directly use ODK-collected blobs in S3-compatible storage
So that I can build public dashboards

As an ODK project manager
I want to browse ODK-collected blobs in S3-compatible storage
So that I can download just the ones I need
@matthew-white matthew-white added enhancement New feature or behavior backend Requires a change to the API server labels Jan 11, 2024
@lognaturel lognaturel linked a pull request Mar 1, 2024 that will close this issue
Open
@alxndrsn
Copy link
Contributor

alxndrsn commented Mar 4, 2024

As a Central administrator

Does this user have access to the server CLI scripts, or only the web frontend?

@lognaturel
Copy link
Member

Does this user have access to the server CLI scripts

Yes! I guess we should say Central system administrator.

@matthew-white matthew-white added the needs testing Needs manual testing label May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alxndrsn alxndrsn

Labels
backend Requires a change to the API server enhancement New feature or behavior needs testing Needs manual testing
Projects
ODK Central
Status: ✏️ in progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support S3-compatible blob storage alxndrsn/odk-central-backend
3 participants
@alxndrsn @lognaturel @matthew-white