AirbyteTriggerSyncOperator¶

Use the AirbyteTriggerSyncOperator to trigger an existing ConnectionId sync job in Airbyte.

Warning

This operator triggers a synchronization job in Airbyte. If triggered again, this operator does not guarantee idempotency. You must be aware of the source (database, API, etc) you are updating/sync and the method applied to perform the operation in Airbyte.

Using the Operator¶

The AirbyteTriggerSyncOperator requires the connection_id this is the uuid identifier create in Airbyte between a source and destination synchronization job. Use the airbyte_conn_id parameter to specify the Airbyte connection to use to connect to your account.

Airbyte offers a single method to authenticate for Cloud and OSS users. You need to provide the client_id and client_secret to authenticate with the Airbyte server.

You can trigger a synchronization job in Airflow in two ways with the Operator. The first one is a synchronous process. This Operator will initiate the Airbyte job, and the Operator manages the job status. Another way is to use the flag async = True so the Operator only triggers the job and returns the job_id, passed to the AirbyteSensor.

An example using the synchronous way:

tests/system/airbyte/example_airbyte_trigger_job.py[source]

    sync_source_destination = AirbyteTriggerSyncOperator(
        task_id="airbyte_sync_source_dest_example",
        connection_id=CONN_ID,
    )

An example using the async way:

tests/system/airbyte/example_airbyte_trigger_job.py[source]

    async_source_destination = AirbyteTriggerSyncOperator(
        task_id="airbyte_async_source_dest_example",
        connection_id=CONN_ID,
        asynchronous=True,
    )

    airbyte_sensor = AirbyteJobSensor(
        task_id="airbyte_sensor_source_dest_example",
        airbyte_job_id=async_source_destination.output,
    )