Sensors

Sensors are a special type of Operator that are designed to do exactly one thing - wait for something to occur. It can be time-based, or waiting for a file, or an external event, but all they do is wait until something happens, and then succeed so their downstream tasks can run.

Because they are primarily idle, Sensors have three different modes of running so you can be a bit more efficient about using them:

  • poke (default): The Sensor takes up a worker slot for its entire runtime

  • reschedule: The Sensor takes up a worker slot only when it is checking, and sleeps for a set duration between checks

  • smart sensor: There is a single centralized version of this Sensor that batches all executions of it

The poke and reschedule modes can be configured directly when you instantiate the sensor; generally, the trade-off between them is latency. Something that is checking every second should be in poke mode, while something that is checking every minute should be in reschedule mode.

Smart Sensors take a bit more setup; for more information on them, see Smart Sensors.

Much like Operators, Airflow has a large set of pre-built Sensors you can use, both in core Airflow as well as via our providers system.

Was this entry helpful?