redshift optimization #4775
redshift optimization #4775
Conversation
| unique_constraints_clean = [ | ||
| f'{self.clean_column_name(col)}' | ||
| for col in unique_constraints | ||
| ] | ||
|
|
||
| drop_duplicate_records_from_temp = (f'DELETE FROM {full_table_name_temp} ' | ||
| f'where ({", ".join(unique_constraints_clean)},_mage_created_at) ' |
There was a problem hiding this comment.
Can we capitalize the WHERE please?
| f'where ({", ".join(unique_constraints_clean)}) ' | ||
| f'in (SELECT {", ".join(unique_constraints_clean)} ' | ||
| f'from {full_table_name_temp})') |
There was a problem hiding this comment.
Can we capitalize the SQL keywords please?
| drop_duplicate_records_from_temp, | ||
| delete_records_from_full_table, | ||
| insert_records_from_temp_table, | ||
| truncate_records_from_temp_table |
There was a problem hiding this comment.
Add trailing comma please.
| f'from {full_table_name_temp})') | ||
|
|
||
| insert_records_from_temp_table = f'INSERT INTO {full_table_name} SELECT * FROM {full_table_name_temp}' | ||
| truncate_records_from_temp_table = f'TRUNCATE table {full_table_name_temp}' |
|
can you describe what tests you ran to verify this change works? |
|
I have done testing below |
|
@dy46 is there any suggestions here ? |
| build_create_table_command( | ||
| column_type_mapping=self.column_type_mapping(schema), | ||
| columns=schema['properties'].keys(), | ||
| full_table_name=f'{temp_table_name}', |
There was a problem hiding this comment.
you can just do full_table_name=temp_table_name here
| drop_duplicate_records_from_temp = (f'DELETE FROM {full_table_name_temp} ' | ||
| f'WHERE ({", ".join(unique_constraints_clean)},_mage_created_at) ' | ||
| f'IN ( SELECT {", ".join(unique_constraints_clean)}, _mage_created_at ' | ||
| f'FROM ( SELECT {", ".join(unique_constraints_clean)}, _mage_created_at, ROW_NUMBER() ' | ||
| f'OVER (PARTITION BY {", ".join(unique_constraints_clean)} ' | ||
| f'ORDER BY _mage_created_at DESC) as row_num ' | ||
| f'FROM {full_table_name_temp} ) AS subquery_alias WHERE row_num > 1);') |
There was a problem hiding this comment.
can you share an example of what this query looks like with the variable values filled in?
There was a problem hiding this comment.
@dy46 Please find sample query here
DELETE FROM temp_abc
WHERE (id,_mage_created_at)
IN ( SELECT id, _mage_created_at
FROM ( SELECT id, _mage_created_at, ROW_NUMBER()
OVER (PARTITION BY id
ORDER BY _mage_created_at DESC) as row_num
FROM temp_abc ) AS subquery_alias WHERE row_num > 1);
There was a problem hiding this comment.
@dy46 changes done as suggested
| f'VALUES {insert_values}', | ||
| ]), | ||
| ] | ||
| # This is temp as need to know the best way to create table programmatically i will change it properly |
There was a problem hiding this comment.
can you elaborate on this comment? not sure what you mean here
There was a problem hiding this comment.
yeah, my bad will remove this comment
| build_create_table_command( | ||
| column_type_mapping=self.column_type_mapping(schema), | ||
| columns=schema['properties'].keys(), | ||
| full_table_name=temp_table_name, | ||
| if_not_exists=True, | ||
| unique_constraints=unique_constraints, | ||
| use_lowercase=self.use_lowercase, | ||
| ) |
There was a problem hiding this comment.
do we have any logic to delete this temporary table?
There was a problem hiding this comment.
No actually, i guess we don't have should we add this ?
There was a problem hiding this comment.
We should if we are going to be creating it every sync
There was a problem hiding this comment.
actually currently it's tried if table not exists only
There was a problem hiding this comment.
I see. I still think we should delete it at the end of the sync to avoid leaving a temp table in the database.
Description
Checklist
docs/mint.jsoncc: