Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe support use within ephemeral environments #38

Open
PeterJCLaw opened this issue Jan 22, 2024 · 0 comments
Open

Maybe support use within ephemeral environments #38

PeterJCLaw opened this issue Jan 22, 2024 · 0 comments

Comments

@PeterJCLaw
Copy link
Collaborator

PeterJCLaw commented Jan 22, 2024

Currently the summary of usage for this package looks like:

(prod)$ python manage.py devdata_export devdata
(prod)$ tar -czf devdata.tar devdata/
(local)$ scp prod:~/devdata.tar devdata.tar.gz
(local)$ tar -xzf devdata.tar.gz
(local)$ python manage.py devdata_import devdata/

This requires that the user can manage the data shuffling themselves and in turn implicitly requires that both sides of the export & import exist for longer than the django command. For environments such as Kubernetes, where a pod may only exist while a command is running, this pattern is a little harder to use. It's definitely still possible though can require a bit more juggling, especially if trying to script the usage rather than use interactively.

I was wondering whether it would make sense for devdata to provide some more direct support for this sort of use-case.

Approaches I can think of which might help:

  • let devdata export to/import from a (compressed) tar file directly, removing the need to have tar in the target environment (as an implementation, this might e.g: use a local temporary directory rather than reading the archive for each file)
  • maybe allow for that archive to come from a Django storage backed location
  • and/or allow for the import/export folders to be universal paths

Universal paths could be used on their own, though given the number of files which devdata tends to create that's likely to be quite inefficient -- I suspect that upaths would probably be more useful as a path to an archive file rather than to a directory.

Open to other ideas too, including if there's an easy way to use devdata in kubernetes which I've missed (if so, perhaps we could document this too?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant