Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

an alternative approach to the cleanup migration #1695

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

hardwarehuman
Copy link
Collaborator

This illustrates some changes. They work, but there's other issues we need to discuss and investigate before we call it done.

Copy link
Collaborator

@AdamAdHocTeam AdamAdHocTeam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this just uses a CTE instead of a temp table.

The run against the most recent db backup from yesterday was 31 minutes just for the delete.

image

Comment on lines +15 to +22
SELECT
"firstName",
"lastName",
"role",
"grantId",
"programId",
active,
MIN(id) AS first_insert
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the best set of fields to match on, I saw some records that only changed email address, while name remained the same. and others where the name changed and email remained the same. I think email needs to be included for comparison.

)
DELETE FROM "ProgramPersonnel"
WHERE id NOT IN (SELECT first_insert FROM initial_entries);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you using the "in select" syntax vs the using syntax


/* 2. Make a copy of the audit log before cleaning it up */
SET session_replication_role = replica;
CREATE TABLE "ZZarchiveZALProgramPersonnel20230828103020" AS SELECT * FROM "ZALProgramPersonnel";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a real table and have an audit log generated for it. I don't think that is what is intended.

WHERE id NOT IN (SELECT first_insert FROM initial_entries);

/* 2. Make a copy of the audit log before cleaning it up */
SET session_replication_role = replica;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not how to turn off audit log

/* 3. Delete everything we don't want to keep */
DELETE FROM "ProgramPersonnel" WHERE id NOT IN (SELECT "idToKeep" FROM "ProgramPersonnelToKeep");
/* 3. Clean out the huge number of records related to deleted dupes */
DELETE FROM "ZALProgramPersonnel" WHERE data_id NOT IN (SELECT id FROM "ProgramPersonnel");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you using the "in select" syntax vs the using syntax

Base automatically changed from al-ttahub-2043-program-personnel-fixes to main September 1, 2023 16:38
@thewatermethod
Copy link
Collaborator

Should we close this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants