airflow.example_dags.example_dataset_alias

Example DAG for demonstrating the behavior of the DatasetAlias feature in Airflow, including conditional and dataset expression-based scheduling.

Notes on usage:

Turn on all the DAGs.

Before running any DAG, the schedule of the “dataset_alias_example_alias_consumer” DAG will show as “Unresolved DatasetAlias”. This is expected because the dataset alias has not been resolved into any dataset yet.

Once the “dataset_s3_bucket_producer” DAG is triggered, the “dataset_s3_bucket_consumer” DAG should be triggered upon completion. This is because the dataset alias “example-alias” is used to add a dataset event to the dataset “s3://bucket/my-task” during the “produce_dataset_events_through_dataset_alias” task. As the DAG “dataset-alias-consumer” relies on dataset alias “example-alias” which was previously unresolved, the DAG “dataset-alias-consumer” (along with all the DAGs in the same file) will be re-parsed and thus update its schedule to the dataset “s3://bucket/my-task” and will also be triggered.

Module Contents

Functions

produce_dataset_events()

produce_dataset_events_through_dataset_alias(*[, ...])

consume_dataset_event()

consume_dataset_event_from_dataset_alias(*[, inlet_events])

airflow.example_dags.example_dataset_alias.produce_dataset_events()[source]
airflow.example_dags.example_dataset_alias.produce_dataset_events_through_dataset_alias(*, outlet_events=None)[source]
airflow.example_dags.example_dataset_alias.consume_dataset_event()[source]
airflow.example_dags.example_dataset_alias.consume_dataset_event_from_dataset_alias(*, inlet_events=None)[source]

Was this entry helpful?