airflow.example_dags.example_dataset_alias_with_no_taskflow¶
Example DAG for demonstrating the behavior of the DatasetAlias feature in Airflow, including conditional and dataset expression-based scheduling.
Notes on usage:
Turn on all the DAGs.
Before running any DAG, the schedule of the “dataset_alias_example_alias_consumer_with_no_taskflow” DAG will show as “unresolved DatasetAlias”. This is expected because the dataset alias has not been resolved into any dataset yet.
Once the “dataset_s3_bucket_producer_with_no_taskflow” DAG is triggered, the “dataset_s3_bucket_consumer_with_no_taskflow” DAG should be triggered upon completion. This is because the dataset alias “example-alias-no-taskflow” is used to add a dataset event to the dataset “s3://bucket/my-task-with-no-taskflow” during the “produce_dataset_events_through_dataset_alias_with_no_taskflow” task. Also, the schedule of the “dataset_alias_example_alias_consumer_with_no_taskflow” DAG should change to “Dataset” as the dataset alias “example-alias-no-taskflow” is now resolved to the dataset “s3://bucket/my-task-with-no-taskflow” and this DAG should also be triggered.
Functions¶
|
|
|