tests.system.openlineage.example_openlineage_task_loop_dag¶
DAG with a plain task and task loops, some of their tasks retrying and one stopped early by until (requires Airflow 3.4+).
- It checks, for a plain task, tasks, a mapped task and a mapped task group in a loop, and loop gates:
taskInstance’s try_number, state, map_index and loop of every event: loop.index is the iteration, counted from 1, and map_index the mapped position, and a plain task has no loop
task’s loop, the loop around the task: its group_id, max_iterations, has_until, gate_task_id and terminal_task_id, also for a task of a task group inside a loop, and absent for the plain task
loop.region_id is set, without comparing it between events, loop.trajectory is 1, and there is no loop.forked_from_region_id nor loop.resumes_from_index, as no loop is cleared
taskInstance’s expanded_ti_count of the mapped tasks, the size of their expansion in the iteration
taskInstance’s max_tries of the tasks other than the gates, the retries of each task
taskInstance’s id is set on every event, and region_id on the tasks in a loop, without comparing them between events, and region_index is the iteration for a task that isn’t mapped and the map index for a mapped one, while a plain task has neither
the gates’ loop.decision: absent on START; for the loops without until, stop true only at the last iteration and no reason; for the loop with until (whose gate is named after it), the reason it returns and stop true at the iteration it ends the loop, before max_iterations; absent for the plain task
a mapped task group inside a loop: its task’s map_index and region_index are the mapped position, its loop.index the iteration, and its task’s loop.group_id the loop’s, unlike its own task_group, whose unique_group_id is the mapped group’s, while for a task directly in a loop it is the loop’s
the loop groups’ max_iterations, has_until, gate_task_id and terminal_task_id in the taskGroups of the Dag’s job facet, and never in the task’s task_group: in task events, only task’s loop describes the loop
the job facet’s is_mapped, is_loop and parse_time_mapped_count of each group, the retries of each task but the gates, and the task_group, loop_group, is_mapped and parse_time_mapped_count (2 for the mapped task group expanded over a literal list, unknown for the task expanded over an upstream result) of plain_task, accumulate.increment, process_batch.process and process_batch.fetch.download
taskInstance’s dag_version_id of every event
a retried task emits START and FAIL for each failed try, then START and COMPLETE for the last one
every execution (iteration, map index, try) has its own runId, shared by its START and COMPLETE/FAIL (the runId check of OpenLineageTestOperator is enabled by default)
every task event’s identity, as system.openlineage.expected_events describes it, and each task in the Dag’s job facet