AirbyteTriggerSyncOperator¶

Use the AirbyteTriggerSyncOperator to trigger an existing ConnectionId sync job in Airbyte.

Warning

This operator triggers a synchronization job in Airbyte. If triggered again, this operator does not guarantee idempotency. You must be aware of the source (database, API, etc) you are updating/sync and the method applied to perform the operation in Airbyte.

Using the Operator¶

The AirbyteTriggerSyncOperator requires the connection_id this is the uuid identifier create in Airbyte between a source and destination synchronization job. Use the airbyte_conn_id parameter to specify the Airbyte connection to use to connect to your account.

Airbyte currently supports two different API’s. The first one is the Config API which is specifically used for Open Source Airbyte Instances. The second is the Cloud API which is used for the Airbyte Cloud Service. If you are using Airbyte’s Cloud service, then you will need to specify api_type="cloud" as part of the Operator’s parameters.

You can trigger a synchronization job in Airflow in two ways with the Operator. The first one is a synchronous process. This Operator will initiate the Airbyte job, and the Operator manages the job status. Another way is to use the flag async = True so the Operator only triggers the job and returns the job_id, passed to the AirbyteSensor.

An example using the synchronous way:

tests/system/providers/airbyte/example_airbyte_trigger_job.py[source]

    sync_source_destination = AirbyteTriggerSyncOperator(
        task_id="airbyte_sync_source_dest_example",
        connection_id=CONN_ID,
    )

An example using the async way:

tests/system/providers/airbyte/example_airbyte_trigger_job.py[source]

    async_source_destination = AirbyteTriggerSyncOperator(
        task_id="airbyte_async_source_dest_example",
        connection_id=CONN_ID,
        asynchronous=True,
    )

    airbyte_sensor = AirbyteJobSensor(
        task_id="airbyte_sensor_source_dest_example",
        airbyte_job_id=async_source_destination.output,
    )