A DAG’s scheduling strategy is determined by its internal “timetable”. This timetable can be created by specifying the DAG’s schedule_interval argument, as described in DAG Run, or by passing a timetable argument directly. The timetable also dictates the data interval and the logical time of each run created for the DAG.

Cron expressions and timedeltas are still supported (using CronDataIntervalTimetable and DeltaDataIntervalTimetable under the hood respectively), however, there are situations where they cannot properly express the schedule. Some examples are:

  • Data intervals with “holes” between. (Instead of continuous, as both the cron expression and timedelta schedules represent.)

  • Run tasks at different times each day. For example, an astronomer may find it useful to run a task at dawn to process data collected from the previous night-time period.

  • Schedules not following the Gregorian calendar. For example, create a run for each month in the Traditional Chinese Calendar. This is conceptually similar to the sunset case above, but for a different time scale.

  • Rolling windows, or overlapping data intervals. For example, one may want to have a run each day, but make each run cover the period of the previous seven days. It is possible to “hack” this with a cron expression, but a custom data interval would be a more natural representation.

As such, Airflow allows for custom timetables to be written in plugins and used by DAGs. An example demonstrating a custom timetable can be found in the Customizing DAG Scheduling with Timetables how-to guide.

Was this entry helpful?