(feat) [nan-591] insert into bigQuery #1903
Conversation
|
Can we start making dedicated packages for this kind of functionality instead of adding it into |
|
I wonder if there's a way to flag or exclude test data. Random idea: accounts created with a @nango.dev email are flagged as test accounts in our main DB & excluded from tracking? |
…-implement-data-ingestion
| insert(row: RunScriptRow): void; | ||
| } | ||
|
|
||
| interface RunScriptRow { |
There was a problem hiding this comment.
Don't love duplicated types but given our setup this is needed for now
There was a problem hiding this comment.
Looks good (can't really test the code) 👌🏻
Maybe we can use optionalDependencies to avoid code duplication, so it would work locally with Typescript and there will be no ref about the type in the code after compilation.
Also it's missing on tsconfig.build.json
| content: string; | ||
| runTimeInSeconds: number; | ||
| createdAt: number; | ||
| } |
There was a problem hiding this comment.
not a blocker: the schema is not linked to the type, which could lead to desync
const fields: { name: keyof RunScriptRow; type: 'STRING' | 'INTEGER' }[] = [
{ name: 'accountId', type: 'STRING' },
{ name: 'connectionId', type: 'STRING' }
];
| } | ||
| } | ||
|
|
||
| public async insert(data: RunScriptRow, tableName?: string) { |
There was a problem hiding this comment.
Since we are not enforcing type and schema matching, should insert simply accept any record<string, string|number>?
There was a problem hiding this comment.
Won't work as then there would be a type mismatch between the two.
Describe your changes
Add in BigQuery data ingestion from sync, actions, and webhooks
Issue ticket number and link
NAN-591
Checklist before requesting a review (skip if just adding/editing APIs & templates)