New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ability to update object with sync mode incremental_deduped_history
#70
Comments
I think this may be related to one I opened as well: #31 |
Hi @ThomasRooney, @terencecho, would you be able to help with this one? Provider is quite unusable in current state. Looking at the provider implementation, features of speakeasyapi.dev it looks as if it wasn't possible to define conditional parameter for request to Airbyte API, meaning Airbyte API would need to be adjusted to accept or ignore primary key parameter instead of throwing error when source already defined it. Anyone can confirm? Would that kind of change in Airbyte API be even possible? |
@necsord I can confirm your understanding. Currently Speakeasy's Provider generator assumes an API described in an OpenAPI spec is mostly idempotent. What this means in this case is that for all resource updates, we create an API request with the intersection of the full set of information that can be derived from resource state and is available in an UPDATE request (i.e. the operation with I think what needs to happen here is either:
If [1], or [2] is done with changes made to https://raw.githubusercontent.com/airbytehq/terraform-provider-airbyte/main/airbyte.yaml, I'm personally not sure what the best approach is. Speakeasy does its best to infer a terraform provider that matches the API as described by the OpenAPI spec: and where there is behaviour not really described by the OpenAPI specification (e.g. sensitive fields, hoisting, reconciliation of API requests/responses to a resource, runtime validations), to augment the OpenAPI spec with annotations that describe the desired behaviour. However we're not experts on the underlying API. Will defer to @terencecho to help guide us forward. |
Thanks @ThomasRooney for your input. Re-creating connection is not an option as it would re-trigger whole sync from scratch which takes hours to complete. I'm not sure if it's possible to implement option 2 as depending on source configuration there should be a different Looking forward to @terencecho input 🙏🏻 We're also considering creating separate incremental resources configuration with dedicated |
The canonical way of describing these is a discriminated It's a little more complicated than what exists now in the API, but I think that would work to describe the API better |
@ThomasRooney Thanks for your input, unfortunately I did not have time to check it up sooner.
Source: PATCH /connections/{connectionId} API documentation I'm not seeing a good discriminator field as I would need information about source, whether it supplies the Change with API would need to be raised separately within airbytehq/airbyte issues? |
Same problem here, even with |
For now can be workaround using lifecycle block
|
@nurikk-sa What would happen when you're modifying, adding new streams to existing connection? |
no drift will happen due to order change, at least it will give some time until this issue is fixed |
@nurikk-sa Brilliant workaround for the ordering changes. Adding/removing streams to an existing connection (issue #70) would still require a delete/recreate of the resource right? |
Did not work for me im still getting this error. *edit i just put primary to [] and it worked |
thank you @fabianboerner that was the only fix that worked for me! to clarify, adjust the configurations like so, to make updates on the streams possible
the downside is that every |
Yes but if you want to setup a connection where you can’t define all streams because they maybe have variable collections in mongo db you will have the same issue again. The only solution I had here was to set the connection to be recreated always but gui user will lose their setup they maybe had done later. It makes the whole thing sometimes tedious. Also if you didn’t have had setup the streams via terraform it will not ignore configuration[“streams”] and the default value is always full overwrite for stream update type wich works within airbyte but not against the api that throws an error. one thing I could life with was the changes terraform would show that are not applied. |
When using sync mode
incremental_deduped_history
, connection updates are not possible. Recreating the connection is required to make changes.Relates to:
The text was updated successfully, but these errors were encountered: