Concepts¶
Here you can find detailed documentation about each one of Airflow’s core concepts and how to use them, as well as a high-level architectural overview.
Architecture
Workloads
- DAGs
- Tasks
- Operators
- Dynamic Task Mapping
- Simple mapping
- Repeated Mapping
- Constant parameters
- Mapping over multiple parameters
- Task-generated Mapping
- Mapping with non-TaskFlow operators
- Mixing TaskFlow and classic operators
- Assigning multiple parameters to a non-TaskFlow operator
- Filtering items from an expanded task
- Transforming mapped data
- Combining upstream data (aka “zipping”)
- What data types can be expanded?
- How do templated fields and mapped arguments interact?
- Placing limits on mapped tasks
- Automatically skipping zero-length maps
- Sensors
- Data-aware scheduling
- Deferrable Operators & Triggers
- TaskFlow
- Executor
- Scheduler
- DAG File Processing
- Pools
- Timetables
- Priority Weights
- Cluster Policies
Communication