Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task]: Setup the transformation process script #1745

Closed
2 tasks
chouinar opened this issue Apr 16, 2024 · 0 comments · Fixed by #1794
Closed
2 tasks

[Task]: Setup the transformation process script #1745

chouinar opened this issue Apr 16, 2024 · 0 comments · Fixed by #1794
Assignees
Labels
project: grants.gov Grants.gov Modernization tickets

Comments

@chouinar
Copy link
Collaborator

Summary

Just setup the script and transform a single table:
https://app.gitbook.com/o/cFcvhi6d0nlLyH2VzVgn/s/v1V0jIH7mb7Yb3jlNrgk/engineering/learnings/opportunity-endpoint-data-model#elt-transformations

Acceptance criteria

  • Setup the script with basic logging / metrics
  • Transforms just the opportunity table (other tables in later tickets)
@chouinar chouinar added the project: grants.gov Grants.gov Modernization tickets label Apr 16, 2024
@chouinar chouinar self-assigned this Apr 16, 2024
chouinar added a commit that referenced this issue May 3, 2024
…pportunity (#1794)

## Summary
Fixes #1745

### Time to review: __10 mins__

## Changes proposed
Setup the transformation process script structure

Implement some shared utilities that subsequent PRs will use

Implement the transformation logic for opportunities

## Context for reviewers
A lot of setup in this PR, a lot that can be reused in the subsequent
PRs to add transformations for the other sets of tables. Tried to make
sure those would require refactoring or pulling out implementation
details by setting up utils like the timestamp conversions + initial
query to the DB to fetch the transforming records.

As far as the implementation goes, determining what needs to be
transformed is pretty simple - the `transformed_at` column is null.
There is then a second column that says whether the record should be
deleted or not. When we query the staging tables, we also join with the
relevant table we'll transform to (opportunity in this case), that way
we already have the "source" and "destination" records and just need to
modify the destination record (or create it if its an insert).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
project: grants.gov Grants.gov Modernization tickets
Projects
2 participants