Example DAG for demonstrating behavior of Datasets feature.

Notes on usage:

Turn on all the dags.

DAG dataset_produces_1 should run because it’s on a schedule.

After dataset_produces_1 runs, dataset_consumes_1 should be triggered immediately because its only dataset dependency is managed by dataset_produces_1.

No other dags should be triggered. Note that even though dataset_consumes_1_and_2 depends on the dataset in dataset_produces_1, it will not be triggered until dataset_produces_2 runs (and dataset_produces_2 is left with no schedule so that we can trigger it manually).

Next, trigger dataset_produces_2. After dataset_produces_2 finishes, dataset_consumes_1_and_2 should run.

Dags dataset_consumes_1_never_scheduled and dataset_consumes_unknown_never_scheduled should not run because they depend on datasets that never get updated.

Module Contents


Was this entry helpful?