Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[catalog-server] Ensure DB partitioning for tests and staging purposes #1351

Open
justinfagnani opened this issue Oct 6, 2022 · 3 comments · Fixed by #1359
Open

[catalog-server] Ensure DB partitioning for tests and staging purposes #1351

justinfagnani opened this issue Oct 6, 2022 · 3 comments · Fixed by #1359
Labels
New Site Issues for the rewrite of the site and catalog.

Comments

@justinfagnani
Copy link
Collaborator

We want to be able to partition the Firestore database for running multiple tests agains the same database in parallel, and to potentially support multi-tenancy, different environments in the same GCP project, etc.

To do this we need to ensure that each collection has a unique name per environment. We do this now with a namespace suffix on the packages collection, but this isn't quite good enough because collection group queries could be performed on other collections. So we need to make sure that .collection() and .collectionGroup() are never called directly, but called through utility methods that add the namespace suffix.

We also need to update the design doc to describe this pattern.

@justinfagnani justinfagnani added the New Site Issues for the rewrite of the site and catalog. label Oct 6, 2022
@justinfagnani
Copy link
Collaborator Author

There's a problem with expressing namespace as collection suffixes: indices. Indices must be created via the console or CLI, and take a few minutes to setup even on empty databases. Indices are also configured by collection IDs, and are required for collection group queries.

The other approach we can take is to add a namespace field to every document and always filter on that field.

@justinfagnani
Copy link
Collaborator Author

And a problem with using a field is that we have a few places where we get a document by building a reference with known document IDs, and not by querying. We need collection suffixes to keep references unique.

So, we can do this:

  • For collections with synthetic document IDs, like customElements, we use a namespace document field
  • For collections with natural document IDs, like packages, we use a collection name suffix.

@justinfagnani
Copy link
Collaborator Author

This wasn't fully fixed yet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Site Issues for the rewrite of the site and catalog.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant