airbyte-config-api-client

Airbyte Configuration API https://airbyte.io.

This API is a collection of HTTP RPC-style methods. While it is not a REST API, those familiar with REST should find the conventions of this API recognizable.

Here are some conventions that this API follows:

All endpoints are http POST methods.
All endpoints accept data via application/json request bodies. The API does not accept any data via query params.
The naming convention for endpoints is: localhost:8000/{VERSION}/{METHOD_FAMILY}/{METHOD_NAME} e.g. localhost:8000/v1/connections/create.
For all update methods, the whole object must be passed in, even the fields that did not change.

Authentication (OSS):

When authenticating to the Configuration API, you must use Basic Authentication by setting the Authentication Header to Basic and base64 encoding the username and password (which are airbyte and password by default - so base64 encoding airbyte:password results in YWlyYnl0ZTpwYXNzd29yZA==). So the full header reads 'Authorization': \"Basic YWlyYnl0ZTpwYXNzd29yZA==\"

This Python package is automatically generated by the OpenAPI Generator project:

API version: 1.0.0
Package version: 1.0.0
Build package: org.openapitools.codegen.languages.PythonClientCodegen

Requirements.

Python >=3.7

Migration from other generators like python and python-legacy

Changes

This generator uses spec case for all (object) property names and parameter names.
- So if the spec has a property name like camelCase, it will use camelCase rather than camel_case
- So you will need to update how you input and read properties to use spec case
Endpoint parameters are stored in dictionaries to prevent collisions (explanation below)
- So you will need to update how you pass data in to endpoints
Endpoint responses now include the original response, the deserialized response body, and (todo)the deserialized headers
- So you will need to update your code to use response.body to access deserialized data
All validated data is instantiated in an instance that subclasses all validated Schema classes and Decimal/str/list/tuple/frozendict/NoneClass/BoolClass/bytes/io.FileIO
- This means that you can use isinstance to check if a payload validated against a schema class
- This means that no data will be of type None/True/False
  - ingested None will subclass NoneClass
  - ingested True will subclass BoolClass
  - ingested False will subclass BoolClass
  - So if you need to check is True/False/None, instead use instance.is_true_oapg()/.is_false_oapg()/.is_none_oapg()
All validated class instances are immutable except for ones based on io.File
- This is because if properties were changed after validation, that validation would no longer apply
- So no changing values or property values after a class has been instantiated
String + Number types with formats
- String type data is stored as a string and if you need to access types based on its format like date, date-time, uuid, number etc then you will need to use accessor functions on the instance
- type string + format: See .as_date_oapg, .as_datetime_oapg, .as_decimal_oapg, .as_uuid_oapg
- type number + format: See .as_float_oapg, .as_int_oapg
- this was done because openapi/json-schema defines constraints. string data may be type string with no format keyword in one schema, and include a format constraint in another schema
- So if you need to access a string format based type, use as_date_oapg/as_datetime_oapg/as_decimal_oapg/as_uuid_oapg
- So if you need to access a number format based type, use as_int_oapg/as_float_oapg
Property access on AnyType(type unset) or object(dict) schemas
- Only required keys with valid python names are properties like .someProp and have type hints
- All optional keys may not exist, so properties are not defined for them
- One can access optional values with dict_instance['optionalProp'] and KeyError will be raised if it does not exist
- Use get_item_oapg if you need a way to always get a value whether or not the key exists
  - If the key does not exist, schemas.unset is returned from calling dict_instance.get_item_oapg('optionalProp')
  - All required and optional keys have type hints for this method, and @typing.overload is used
  - A type hint is also generated for additionalProperties accessed using this method
- So you will need to update you code to use some_instance['optionalProp'] to access optional property and additionalProperty values
The location of the api classes has changed
- Api classes are located in your_package.apis.tags.some_api
- This change was made to eliminate redundant code generation
- Legacy generators generated the same endpoint twice if it had > 1 tag on it
- This generator defines an endpoint in one class, then inherits that class to generate apis by tags and by paths
- This change reduces code and allows quicker run time if you use the path apis
  - path apis are at your_package.apis.paths.some_path
- Those apis will only load their needed models, which is less to load than all of the resources needed in a tag api
- So you will need to update your import paths to the api classes

Why are Oapg and _oapg used in class and method names?

Classes can have arbitrarily named properties set on them Endpoints can have arbitrary operationId method names set For those reasons, I use the prefix Oapg and _oapg to greatly reduce the likelihood of collisions on protected + public classes/methods. oapg stands for OpenApi Python Generator.

Object property spec case

This was done because when payloads are ingested, they can be validated against N number of schemas. If the input signature used a different property name then that has mutated the payload. So SchemaA and SchemaB must both see the camelCase spec named variable. Also it is possible to send in two properties, named camelCase and camel_case in the same payload. That use case should be support so spec case is used.

Parameter spec case

Parameters can be included in different locations including:

query
path
header
cookie

Any of those parameters could use the same parameter names, so if every parameter was included as an endpoint parameter in a function signature, they would collide. For that reason, each of those inputs have been separated out into separate typed dictionaries:

query_params
path_params
header_params
cookie_params

So when updating your code, you will need to pass endpoint parameters in using those dictionaries.

Endpoint responses

Endpoint responses have been enriched to now include more information. Any response reom an endpoint will now include the following properties: response: urllib3.HTTPResponse body: typing.Union[Unset, Schema] headers: typing.Union[Unset, TODO] Note: response header deserialization has not yet been added

Installation & Usage

pip install

If the python package is hosted on a repository, you can install directly using:

pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git

(you may need to run pip with root permission: sudo pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git)

Then import the package:

import airbyte_config_api_client

Setuptools

Install via Setuptools.

python setup.py install --user

(or sudo python setup.py install to install the package for all users)

Then import the package:

import airbyte_config_api_client

Getting Started

Please follow the installation procedure and then run the following:

import time
import airbyte_config_api_client
from pprint import pprint
from airbyte_config_api_client.apis.tags import attempt_api
from airbyte_config_api_client.model.internal_operation_result import InternalOperationResult
from airbyte_config_api_client.model.save_attempt_sync_config_request_body import SaveAttemptSyncConfigRequestBody
from airbyte_config_api_client.model.save_stats_request_body import SaveStatsRequestBody
from airbyte_config_api_client.model.set_workflow_in_attempt_request_body import SetWorkflowInAttemptRequestBody
# Defining the host is optional and defaults to http://localhost:8000/api
# See configuration.py for a list of all supported configuration parameters.
configuration = airbyte_config_api_client.Configuration(
    host = "http://localhost:8000/api"
    username = "username"
    password = "password"
)


# Enter a context with an instance of the API client
with airbyte_config_api_client.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = attempt_api.AttemptApi(api_client)
    save_stats_request_body = SaveStatsRequestBody(
        job_id=1,
        attempt_number=1,
        stats=AttemptStats(
            records_emitted=1,
            bytes_emitted=1,
            state_messages_emitted=1,
            records_committed=1,
            estimated_records=1,
            estimated_bytes=1,
        ),
        stream_stats=[
            AttemptStreamStats(
                stream_name="stream_name_example",
                stream_namespace="stream_namespace_example",
                stats=AttemptStats(),
            )
        ],
    ) # SaveStatsRequestBody | 

    try:
        # For worker to set sync stats of a running attempt.
        api_response = api_instance.save_stats(save_stats_request_body)
        pprint(api_response)
    except airbyte_config_api_client.ApiException as e:
        print("Exception when calling AttemptApi->save_stats: %s\n" % e)

Documentation for API Endpoints

All URIs are relative to http://localhost:8000/api

Class	Method	HTTP request	Description
AttemptApi	save_stats	post /v1/attempt/save_stats	For worker to set sync stats of a running attempt.
AttemptApi	save_sync_config	post /v1/attempt/save_sync_config	For worker to save the AttemptSyncConfig for an attempt.
AttemptApi	set_workflow_in_attempt	post /v1/attempt/set_workflow_in_attempt	For worker to register the workflow id in attempt.
ConnectionApi	create_connection	post /v1/connections/create	Create a connection between a source and a destination
ConnectionApi	delete_connection	post /v1/connections/delete	Delete a connection
ConnectionApi	get_connection	post /v1/connections/get	Get a connection
ConnectionApi	list_all_connections_for_workspace	post /v1/connections/list_all	Returns all connections for a workspace, including deleted connections.
ConnectionApi	list_connections_for_workspace	post /v1/connections/list	Returns all connections for a workspace.
ConnectionApi	reset_connection	post /v1/connections/reset	Reset the data for the connection. Deletes data generated by the connection in the destination. Resets any cursors back to initial state.
ConnectionApi	search_connections	post /v1/connections/search	Search connections
ConnectionApi	sync_connection	post /v1/connections/sync	Trigger a manual sync of the connection
ConnectionApi	update_connection	post /v1/connections/update	Update a connection
DestinationApi	check_connection_to_destination	post /v1/destinations/check_connection	Check connection to the destination
DestinationApi	check_connection_to_destination_for_update	post /v1/destinations/check_connection_for_update	Check connection for a proposed update to a destination
DestinationApi	clone_destination	post /v1/destinations/clone	Clone destination
DestinationApi	create_destination	post /v1/destinations/create	Create a destination
DestinationApi	delete_destination	post /v1/destinations/delete	Delete the destination
DestinationApi	get_destination	post /v1/destinations/get	Get configured destination
DestinationApi	list_destinations_for_workspace	post /v1/destinations/list	List configured destinations for a workspace
DestinationApi	search_destinations	post /v1/destinations/search	Search destinations
DestinationApi	update_destination	post /v1/destinations/update	Update a destination
DestinationDefinitionApi	create_custom_destination_definition	post /v1/destination_definitions/create_custom	Creates a custom destinationDefinition for the given workspace
DestinationDefinitionApi	delete_destination_definition	post /v1/destination_definitions/delete	Delete a destination definition
DestinationDefinitionApi	get_destination_definition	post /v1/destination_definitions/get	Get destinationDefinition
DestinationDefinitionApi	get_destination_definition_for_workspace	post /v1/destination_definitions/get_for_workspace	Get a destinationDefinition that is configured for the given workspace
DestinationDefinitionApi	grant_destination_definition_to_workspace	post /v1/destination_definitions/grant_definition	grant a private, non-custom destinationDefinition to a given workspace
DestinationDefinitionApi	list_destination_definitions	post /v1/destination_definitions/list	List all the destinationDefinitions the current Airbyte deployment is configured to use
DestinationDefinitionApi	list_destination_definitions_for_workspace	post /v1/destination_definitions/list_for_workspace	List all the destinationDefinitions the given workspace is configured to use
DestinationDefinitionApi	list_latest_destination_definitions	post /v1/destination_definitions/list_latest	List the latest destinationDefinitions Airbyte supports
DestinationDefinitionApi	list_private_destination_definitions	post /v1/destination_definitions/list_private	List all private, non-custom destinationDefinitions, and for each indicate whether the given workspace has a grant for using the definition. Used by admins to view and modify a given workspace's grants.
DestinationDefinitionApi	revoke_destination_definition_from_workspace	post /v1/destination_definitions/revoke_definition	revoke a grant to a private, non-custom destinationDefinition from a given workspace
DestinationDefinitionApi	update_destination_definition	post /v1/destination_definitions/update	Update destinationDefinition
DestinationDefinitionSpecificationApi	get_destination_definition_specification	post /v1/destination_definition_specifications/get	Get specification for a destinationDefinition
DestinationOauthApi	complete_destination_o_auth	post /v1/destination_oauths/complete_oauth	Given a destination def ID generate an access/refresh token etc.
DestinationOauthApi	get_destination_o_auth_consent	post /v1/destination_oauths/get_consent_url	Given a destination connector definition ID, return the URL to the consent screen where to redirect the user to.
DestinationOauthApi	set_instancewide_destination_oauth_params	post /v1/destination_oauths/oauth_params/create	Sets instancewide variables to be used for the oauth flow when creating this destination. When set, these variables will be injected into a connector's configuration before any interaction with the connector image itself. This enables running oauth flows with consistent variables e.g: the company's Google Ads developer_token, client_id, and client_secret without the user having to know about these variables.
HealthApi	get_health_check	get /v1/health	Health Check
InternalApi	create_or_update_state	post /v1/state/create_or_update	Create or update the state for a connection.
InternalApi	get_attempt_normalization_statuses_for_job	post /v1/jobs/get_normalization_status	Get normalization status to determine if we can bypass normalization phase
InternalApi	save_stats	post /v1/attempt/save_stats	For worker to set sync stats of a running attempt.
InternalApi	save_sync_config	post /v1/attempt/save_sync_config	For worker to save the AttemptSyncConfig for an attempt.
InternalApi	set_workflow_in_attempt	post /v1/attempt/set_workflow_in_attempt	For worker to register the workflow id in attempt.
InternalApi	write_discover_catalog_result	post /v1/sources/write_discover_catalog_result	Should only called from worker, to write result from discover activity back to DB.
JobsApi	cancel_job	post /v1/jobs/cancel	Cancels a job
JobsApi	get_attempt_normalization_statuses_for_job	post /v1/jobs/get_normalization_status	Get normalization status to determine if we can bypass normalization phase
JobsApi	get_job_debug_info	post /v1/jobs/get_debug_info	Gets all information needed to debug this job
JobsApi	get_job_info	post /v1/jobs/get	Get information about a job
JobsApi	get_job_info_light	post /v1/jobs/get_light	Get information about a job excluding attempt info and logs
JobsApi	get_last_replication_job	post /v1/jobs/get_last_replication_job
JobsApi	list_jobs_for	post /v1/jobs/list	Returns recent jobs for a connection. Jobs are returned in descending order by createdAt.
LogsApi	get_logs	post /v1/logs/get	Get logs
NotificationsApi	try_notification_config	post /v1/notifications/try	Try sending a notifications
OpenapiApi	get_open_api_spec	get /v1/openapi	Returns the openapi specification
OperationApi	check_operation	post /v1/operations/check	Check if an operation to be created is valid
OperationApi	create_operation	post /v1/operations/create	Create an operation to be applied as part of a connection pipeline
OperationApi	delete_operation	post /v1/operations/delete	Delete an operation
OperationApi	get_operation	post /v1/operations/get	Returns an operation
OperationApi	list_operations_for_connection	post /v1/operations/list	Returns all operations for a connection.
OperationApi	update_operation	post /v1/operations/update	Update an operation
SchedulerApi	execute_destination_check_connection	post /v1/scheduler/destinations/check_connection	Run check connection for a given destination configuration
SchedulerApi	execute_source_check_connection	post /v1/scheduler/sources/check_connection	Run check connection for a given source configuration
SchedulerApi	execute_source_discover_schema	post /v1/scheduler/sources/discover_schema	Run discover schema for a given source a source configuration
SourceApi	check_connection_to_source	post /v1/sources/check_connection	Check connection to the source
SourceApi	check_connection_to_source_for_update	post /v1/sources/check_connection_for_update	Check connection for a proposed update to a source
SourceApi	clone_source	post /v1/sources/clone	Clone source
SourceApi	create_source	post /v1/sources/create	Create a source
SourceApi	delete_source	post /v1/sources/delete	Delete a source
SourceApi	discover_schema_for_source	post /v1/sources/discover_schema	Discover the schema catalog of the source
SourceApi	get_most_recent_source_actor_catalog	post /v1/sources/most_recent_source_actor_catalog	Get most recent ActorCatalog for source
SourceApi	get_source	post /v1/sources/get	Get source
SourceApi	list_sources_for_workspace	post /v1/sources/list	List sources for workspace
SourceApi	search_sources	post /v1/sources/search	Search sources
SourceApi	update_source	post /v1/sources/update	Update a source
SourceApi	write_discover_catalog_result	post /v1/sources/write_discover_catalog_result	Should only called from worker, to write result from discover activity back to DB.
SourceDefinitionApi	create_custom_source_definition	post /v1/source_definitions/create_custom	Creates a custom sourceDefinition for the given workspace
SourceDefinitionApi	delete_source_definition	post /v1/source_definitions/delete	Delete a source definition
SourceDefinitionApi	get_source_definition	post /v1/source_definitions/get	Get source
SourceDefinitionApi	get_source_definition_for_workspace	post /v1/source_definitions/get_for_workspace	Get a sourceDefinition that is configured for the given workspace
SourceDefinitionApi	grant_source_definition_to_workspace	post /v1/source_definitions/grant_definition	grant a private, non-custom sourceDefinition to a given workspace
SourceDefinitionApi	list_latest_source_definitions	post /v1/source_definitions/list_latest	List the latest sourceDefinitions Airbyte supports
SourceDefinitionApi	list_private_source_definitions	post /v1/source_definitions/list_private	List all private, non-custom sourceDefinitions, and for each indicate whether the given workspace has a grant for using the definition. Used by admins to view and modify a given workspace's grants.
SourceDefinitionApi	list_source_definitions	post /v1/source_definitions/list	List all the sourceDefinitions the current Airbyte deployment is configured to use
SourceDefinitionApi	list_source_definitions_for_workspace	post /v1/source_definitions/list_for_workspace	List all the sourceDefinitions the given workspace is configured to use
SourceDefinitionApi	revoke_source_definition_from_workspace	post /v1/source_definitions/revoke_definition	revoke a grant to a private, non-custom sourceDefinition from a given workspace
SourceDefinitionApi	update_source_definition	post /v1/source_definitions/update	Update a sourceDefinition
SourceDefinitionSpecificationApi	get_source_definition_specification	post /v1/source_definition_specifications/get	Get specification for a SourceDefinition.
SourceOauthApi	complete_source_o_auth	post /v1/source_oauths/complete_oauth	Given a source def ID generate an access/refresh token etc.
SourceOauthApi	get_source_o_auth_consent	post /v1/source_oauths/get_consent_url	Given a source connector definition ID, return the URL to the consent screen where to redirect the user to.
SourceOauthApi	set_instancewide_source_oauth_params	post /v1/source_oauths/oauth_params/create	Sets instancewide variables to be used for the oauth flow when creating this source. When set, these variables will be injected into a connector's configuration before any interaction with the connector image itself. This enables running oauth flows with consistent variables e.g: the company's Google Ads developer_token, client_id, and client_secret without the user having to know about these variables.
StateApi	create_or_update_state	post /v1/state/create_or_update	Create or update the state for a connection.
StateApi	get_state	post /v1/state/get	Fetch the current state for a connection.
WebBackendApi	get_state_type	post /v1/web_backend/state/get_type	Fetch the current state type for a connection.
WebBackendApi	web_backend_check_updates	post /v1/web_backend/check_updates	Returns a summary of source and destination definitions that could be updated.
WebBackendApi	web_backend_create_connection	post /v1/web_backend/connections/create	Create a connection
WebBackendApi	web_backend_get_connection	post /v1/web_backend/connections/get	Get a connection
WebBackendApi	web_backend_get_workspace_state	post /v1/web_backend/workspace/state	Returns the current state of a workspace
WebBackendApi	web_backend_list_connections_for_workspace	post /v1/web_backend/connections/list	Returns all non-deleted connections for a workspace.
WebBackendApi	web_backend_list_geographies	post /v1/web_backend/geographies/list	Returns available geographies can be selected to run data syncs in a particular geography. The 'auto' entry indicates that the sync will be automatically assigned to a geography according to the platform default behavior. Entries other than 'auto' are two-letter country codes that follow the ISO 3166-1 alpha-2 standard.
WebBackendApi	web_backend_update_connection	post /v1/web_backend/connections/update	Update a connection
WorkspaceApi	create_workspace	post /v1/workspaces/create	Creates a workspace
WorkspaceApi	delete_workspace	post /v1/workspaces/delete	Deletes a workspace
WorkspaceApi	get_workspace	post /v1/workspaces/get	Find workspace by ID
WorkspaceApi	get_workspace_by_connection_id	post /v1/workspaces/get_by_connection_id	Find workspace by connection id
WorkspaceApi	get_workspace_by_slug	post /v1/workspaces/get_by_slug	Find workspace by slug
WorkspaceApi	list_workspaces	post /v1/workspaces/list	List all workspaces registered in the current Airbyte deployment
WorkspaceApi	update_workspace	post /v1/workspaces/update	Update workspace state
WorkspaceApi	update_workspace_feedback	post /v1/workspaces/tag_feedback_status_as_done	Update workspace feedback state
WorkspaceApi	update_workspace_name	post /v1/workspaces/update_name	Update workspace name

Documentation For Models

Documentation For Authorization

Authentication schemes defined for the API:

basicAuth

Type: HTTP basic authentication

Author

contact@airbyte.io

Notes for Large OpenAPI documents

If the OpenAPI document is large, imports in airbyte_config_api_client.apis and airbyte_config_api_client.models may fail with a RecursionError indicating the maximum recursion limit has been exceeded. In that case, there are a couple of solutions:

Solution 1: Use specific imports for apis and models like:

from airbyte_config_api_client.apis.default_api import DefaultApi
from airbyte_config_api_client.model.pet import Pet

Solution 1: Before importing the package, adjust the maximum recursion limit as shown below:

import sys
sys.setrecursionlimit(1500)
import airbyte_config_api_client
from airbyte_config_api_client.apis import *
from airbyte_config_api_client.models import *

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
.openapi-generator		.openapi-generator
airbyte_config_api_client		airbyte_config_api_client
docs		docs
test		test
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.openapi-generator-ignore		.openapi-generator-ignore
.travis.yml		.travis.yml
LICENSE		LICENSE
README.md		README.md
config.yaml		config.yaml
git_push.sh		git_push.sh
requirements.txt		requirements.txt
setup.cfg		setup.cfg
setup.py		setup.py
test-requirements.txt		test-requirements.txt
tox.ini		tox.ini

License

amarnus/airbyte-config-api-python-client

Folders and files

Latest commit

History

Repository files navigation