[Issue #1686] load data from legacy (Oracle) tables to staging tables

[jamesbursa]

Summary

Fixes #1686

Time to review: 10 mins

Changes proposed

• Add class LoadOracleDataTask which loops through each legacy table and loads the data into the corresponding staging table.
• Add a module of functions src.data_migration.load.sql to generate sqlalchemy SQL expressions for the loads.
• Add a script seed_local_legacy_tables.py to generate fake data in the local legacy tables.
• Add unit tests.

Context for reviewers

This is a new batch task which loads data from the legacy (Oracle) tables to staging tables.

It is incremental and handles the following situations:

Situation	Detection logic	Action
New row in source table	Primary key not present in destination table	Insert into destination, set transformed_at=NULL
Updated row in source	Primary key exists in both source & destination AND newer last_upd_date	Update in destination, reset transformed_at=NULL
Deleted row in source	Primary key does not exist in source, AND not is_deleted in destination	Set is_deleted=TRUE, reset transformed_at=NULL
Unchanged	Primary key exists in both source & destination AND last_upd_date is not newer	No write to destination

Additional information

To run the code locally with fake data:

make setup-foreign-tables  # once
make seed-local-legacy-tables  # repeat to add & update data
poetry run python3 -m src.data_migration.load.load_oracle_data_task

[chouinar]

Maybe as a follow-up if we want to improve performance, we could consider getting the count in the same query using RETURNING https://www.postgresql.org/docs/current/dml-returning.html

Did a little bit of testing locally - seems like its doable:

-- Sequence and defined type
CREATE SEQUENCE IF NOT EXISTS api.my_table_id_seq;

-- Table Definition
CREATE TABLE "api"."my_table" (
    "id" int4 NOT NULL DEFAULT nextval('api.my_table_id_seq'::regclass),
    "name" text,
    PRIMARY KEY ("id")

-- Actual query
with rows as (
INSERT INTO api.my_table (id, name)
		VALUES(5, 'bob'), (6, 'joe'), (7, 'steve')
	RETURNING
		id
	)
	select count(*) from rows

This gives 3 in the response.

[chouinar]

One thing I think we should look into for the factories is a way to do this automatically - they mention being able to force the sequence counter based on a startup method in https://factoryboy.readthedocs.io/en/stable/recipes.html#forcing-the-sequence-counter

Basically what you're doing here / what I did in the db-seed-local script, but in the factory itself.

[chouinar]

A few minor suggestions, but looks good to me

[chouinar]

LGTM!

[chouinar]

The commit isn't necessary as it happens in the exit part of the begin method.

Suggested change

	try:
	with self.db_session.begin():
	self.load_data_for_table(table_name)
	self.db_session.commit()
	except Exception:
	logger.exception("table load error", extra={"table": table_name})
	try:
	with self.db_session.begin():
	self.load_data_for_table(table_name)
	except Exception:
	logger.exception("table load error", extra={"table": table_name})

[coilysiren]

rubber stamp to echo Michael's approval 🙏🏼