Remove IndexMap document list #3606
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PBENCH-1315 The production server, with "only" 108,728 indexed datasets (many more still haven't been migrated from the passthrough server), currently claims 84.1Gb of PostgreSQL storage just for the `IndexMap` table. Most of this consists of a list of each Opensearch document ID in order to allow using bulk update and delete operations to manage the index. This is straining the capacity of our RDU2 PostgreSQL server. As an alternative, this PR removes the document list and instead of the bulk update and delete operations uses `_delete_by_query` and `_update_by_query` searching for documents in the appropriate indices (which we still store in the `IndexMap`) by parent dataset resource ID. Along the way, I noticed that (oops) we were missing the `"authorization"` subdocument in some of our Elasticsearch documents, which would impact the authenticated search API behaviors. And I acted on a deprecation warning for a camelCase template keyword by replacing it with a snake_case alternative. __DRAFT__: I've completed the basic implementation, so that can be reviewed, along with the database model unit test, but the API unit test is still pending. (I'll continue working on that as I have time, but thought I might as well post what I've got so far.)
dbutenhof
added
bug
Server
Indexing
Database
Operations
webbnh
reviewed
Feb 1, 2024
1. Remove the nascent `test_datasets_delete.py` unit test; we have a priority to quickly defer this SQL fix to reduce our load on the RDU2 PostgreSQL. 2. I'd successfully run functional tests, but as I'd wanted to debug within the container I'd been running `runlocal --keep`, which skips the dataset `DELETE` and avoided exposing the fundamental bug that `ElasticBase` doesn't expose `_delete`. (Oops.) 3. Some general cleanup. There are probably some `INFO` log messages that could be removed, but for now I'd prefer that this subsystem remain a bit chatty in case we encounter any problems.
Merged
webbnh
approved these changes
Feb 1, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PBENCH-1315
The production server, with "only" 108,728 indexed datasets (many more still haven't been migrated from the passthrough server), currently claims 84.1Gb of PostgreSQL storage just for the
IndexMaptable. Most of this consists of a list of each Opensearch document ID in order to allow using bulk update and delete operations to manage the index. This is straining the capacity of our RDU2 PostgreSQL server.As an alternative, this PR removes the document list and instead of the bulk update and delete operations uses
_delete_by_queryand_update_by_querysearching for documents in the appropriate indices (which we still store in theIndexMap) by parent dataset resource ID.Along the way, I noticed that (oops) we were missing the
"authorization"subdocument in some of our Elasticsearch documents, which would impact the authenticated search API behaviors. And I acted on a deprecation warning for a camelCase template keyword by replacing it with a snake_case alternative.NOTE: In the interest of expediently deploying a fix for our SQL bloat in RDU2, this is missing unit testing for
updateanddelete, both of which are tested (in indexed and non-indexed cases) by functional tests.