Metrics¶

Airflow can be set up to send metrics to StatsD.

Setup¶

First you must install statsd requirement:

pip install 'apache-airflow[statsd]'

Add the following lines to your configuration file e.g. airflow.cfg

[scheduler]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

Name	Description
<job_name>_start	Number of started <job_name> job, ex. SchedulerJob, LocalTaskJob
<job_name>_end	Number of ended <job_name> job, ex. SchedulerJob, LocalTaskJob
operator_failures_<operator_name>	Operator <operator_name> failures
operator_successes_<operator_name>	Operator <operator_name> successes
ti_failures	Overall task instances failures
ti_successes	Overall task instances successes
zombies_killed	Zombie tasks killed
scheduler_heartbeat	Scheduler heartbeats

Name	Description
collect_dags	Seconds taken to scan and import DAGs
dagbag_import_errors	DAG import errors
dagbag_size	DAG bag size
dag_processing.last_runtime.<dag_file>	Seconds spent processing <dag_file> (in most recent iteration)
dag_processing.last_run.seconds_ago.<dag_file>	Seconds since <dag_file> was last processed
executor.open_slots	Number of open slots on executor
executor.queued_tasks	Number of queued tasks on executor
executor.running_tasks	Number of running tasks on executor
pool.open_slots.<pool_name>	Number of open slots in the pool
pool.used_slots.<pool_name>	Number of used slots in the pool
pool.starving_tasks.<pool_name>	Number of starving tasks in the pool

Name	Description
dagrun.dependency-check.<dag_id>	Seconds taken to check DAG dependencies
dag.<dag_id>.<task_id>.duration	Seconds taken to finish a task
dag.loading-duration.<dag_id>	Seconds taken to load the given DAG
dagrun.duration.success.<dag_id>	Seconds taken for a DagRun to reach success state
dagrun.duration.failed.<dag_id>	Seconds taken for a DagRun to reach failed state
dagrun.schedule_delay.<dag_id>	Seconds of delay between the scheduled DagRun start date and the actual DagRun start date