Python API Reference¶
The DAG is Airflow’s core model that represents a recurring workflow. Check out
DAG for details.
Operators allow for generation of certain types of tasks that become nodes in
the DAG when instantiated. All operators derive from
inherit many attributes and methods that way.
There are 3 main types of operators:
Operators that performs an action, or tell another system to perform an action
Transfer operators move data from one system to another
Sensors are a certain type of operator that will keep running until a certain criterion is met. Examples include a specific file landing in HDFS or S3, a partition appearing in Hive, or a specific time of the day. Sensors are derived from
BaseSensorOperatorand run a poke method at a specified
poke_intervaluntil it returns
All operators are derived from
BaseOperator and acquire much
functionality through inheritance. Since this is the core of the engine,
it’s worth taking the time to understand the parameters of
to understand the primitive features that can be leveraged in your
All sensors are derived from
BaseSensorOperator. All sensors inherit
poke_interval on top of the
All operators are in the following packages:
Hooks are interfaces to external platforms and databases, implementing a common
interface when possible and acting as building blocks for operators. All hooks
are derived from
All hooks are in the following packages:
Executors are the mechanism by which task instances get run. All executors are
All executors are in the following packages:
Models are built on top of the SQLAlchemy ORM Base class, and instances are persisted in the database.
Airflow relies on secrets backends to retrieve
All secrets backends derive from
Custom timetable implementations provide Airflow’s scheduler additional logic to schedule DAG runs in ways not possible with built-in schedule expressions.