Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destinations CDK: CatalogParser sets default namespace #38121

Open
wants to merge 1 commit into
base: edgao/bigquery_new_columns
Choose a base branch
from
Open

Destinations CDK: CatalogParser sets default namespace #38121

wants to merge 1 commit into from
+122 −185

Conversation

edgao
Copy link
Contributor

@edgao edgao commented May 10, 2024 •
edited

  • CatalogParser is now responsible for setting the default namespace on stream objects.
    • This entails a bunch of knock-on changes:
      • All the core async stuff now operates on the original (null) namespace
      • BufferEnqueue also needs to set the default namespace right before handing the message to the queue
      • AsyncStreamConsumer needs to set the default namespace when calling OnCloseFunction
    • I also took the opportunity to make defaultNamespace non-nullable.
  • This allows destinations code to stop thinking about it (i.e. we can remove all our catalog.streams.each(stream.setNamespace) blocks)

closes https://github.com/airbytehq/airbyte-internal-issues/issues/2413

@vercel Vercel
Copy link

vercel bot commented May 10, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview May 23, 2024 8:43pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label May 10, 2024
@edgao Graphite App
Copy link
Contributor Author

edgao commented May 10, 2024 •
edited

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite Graphite

This was referenced May 10, 2024
Draft
Open
Closed
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch 2 times, most recently from 2047aec to 2f4b6ac Compare May 10, 2024 17:33
@gisripa gisripa mentioned this pull request May 10, 2024
Merged
2 tasks
@edgao edgao changed the base branch from master to cdk_generation_id May 10, 2024 19:59
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 2f4b6ac to 08d7792 Compare May 10, 2024 19:59
@edgao edgao mentioned this pull request May 10, 2024
Merged
@edgao edgao changed the title Destinations CDK: CatalogParser extracts generation ID, sets default namespace Destinations CDK: CatalogParser extracts sets default namespace May 10, 2024
@edgao edgao changed the title Destinations CDK: CatalogParser extracts sets default namespace Destinations CDK: CatalogParser sets default namespace May 10, 2024
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch 2 times, most recently from 63d10dc to 55209ad Compare May 10, 2024 20:32
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 55209ad to 7ac6df3 Compare May 10, 2024 22:18
This was referenced May 10, 2024
Merged
Merged
Merged
@gisripa gisripa force-pushed the cdk_generation_id branch 2 times, most recently from 0ded7a7 to 23a237f Compare May 13, 2024 15:42
Base automatically changed from cdk_generation_id to master May 13, 2024 16:59
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 7ac6df3 to 9cb852a Compare May 13, 2024 18:17
@edgao edgao mentioned this pull request May 15, 2024
Closed
2 tasks
@gisripa gisripa mentioned this pull request May 16, 2024
Merged
2 tasks
@gisripa gisripa force-pushed the edgao/catalog_parser_improvements branch from 1386471 to ce689f5 Compare May 21, 2024 17:57
@gisripa gisripa force-pushed the gireesh/bq-refactor-new-intfs branch from 866d3a6 to 796002e Compare May 21, 2024 20:55
@gisripa gisripa force-pushed the edgao/catalog_parser_improvements branch from ce689f5 to 1f37df1 Compare May 21, 2024 20:55
@gisripa gisripa force-pushed the gireesh/bq-refactor-new-intfs branch from 796002e to abdfe71 Compare May 21, 2024 21:11
@gisripa gisripa force-pushed the edgao/catalog_parser_improvements branch from 1f37df1 to f2aa8b6 Compare May 21, 2024 21:12
@edgao edgao force-pushed the gireesh/bq-refactor-new-intfs branch 2 times, most recently from 891944a to aa20464 Compare May 21, 2024 22:01
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from f2aa8b6 to 259a416 Compare May 21, 2024 22:01
@edgao edgao marked this pull request as ready for review May 21, 2024 22:15
@edgao edgao requested review from a team as code owners May 21, 2024 22:15
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch 2 times, most recently from b5f91ac to 4776048 Compare May 22, 2024 17:50
@gisripa gisripa force-pushed the gireesh/bq-refactor-new-intfs branch from aa20464 to c59d16c Compare May 22, 2024 18:55
@gisripa gisripa force-pushed the edgao/catalog_parser_improvements branch from 4776048 to bd00f36 Compare May 22, 2024 18:55
@edgao edgao force-pushed the gireesh/bq-refactor-new-intfs branch from c59d16c to a5ec528 Compare May 22, 2024 20:35
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from bd00f36 to 4d70f49 Compare May 22, 2024 20:36
@gisripa gisripa force-pushed the gireesh/bq-refactor-new-intfs branch from a5ec528 to 30e1e6a Compare May 22, 2024 21:40
@gisripa gisripa requested review from a team as code owners May 22, 2024 21:40
@gisripa gisripa force-pushed the edgao/catalog_parser_improvements branch from 4d70f49 to 09dd1f6 Compare May 22, 2024 21:40
Base automatically changed from gireesh/bq-refactor-new-intfs to master May 22, 2024 22:34
@edgao edgao mentioned this pull request May 23, 2024
Merged
2 tasks
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 09dd1f6 to 3ec532f Compare May 23, 2024 14:51
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 3ec532f to 9f03010 Compare May 23, 2024 20:43
@edgao edgao changed the base branch from master to edgao/bigquery_new_columns May 23, 2024 20:43
@edgao edgao mentioned this pull request May 23, 2024
Draft
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gisripa gisripa gisripa left review comments

@stephane-airbyte stephane-airbyte stephane-airbyte approved these changes

Assignees
No one assigned
Labels
CDK Connector Development Kit
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@edgao @gisripa @stephane-airbyte @octavia-squidington-iii