Command Line Interface¶

Airflow has a very rich command line interface that allows for many types of operation on a DAG, starting services, and supporting development and testing.

usage: airflow [-h]
               {version,initdb,upgradedb,delete_dag,task_state,list_dags,resetdb,create_user,webserver,pool,scheduler,serve_logs,clear,trigger_dag,test,connections,worker,kerberos,pause,task_failed_deps,render,run,list_tasks,backfill,dag_state,variables,flower,unpause}
               ...

Positional Arguments¶

subcommand

Possible choices: version, initdb, upgradedb, delete_dag, task_state, list_dags, resetdb, create_user, webserver, pool, scheduler, serve_logs, clear, trigger_dag, test, connections, worker, kerberos, pause, task_failed_deps, render, run, list_tasks, backfill, dag_state, variables, flower, unpause

sub-command help

Sub-commands:¶

version¶

Show the version

airflow version [-h]

initdb¶

Initialize the metadata database

airflow initdb [-h]

upgradedb¶

Upgrade the metadata database to latest version

airflow upgradedb [-h]

delete_dag¶

Delete all DB records related to the specified DAG

airflow delete_dag [-h] [-y] dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

-y, --yes

Do not prompt to confirm reset. Use with care!

Default: False

task_state¶

Get the status of a task instance

airflow task_state [-h] [-sd SUBDIR] dag_id task_id execution_date

Positional Arguments¶

`dag_id`	The id of the dag
`task_id`	The id of the task
`execution_date`	The execution date of the DAG

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

list_dags¶

List all the DAGs

airflow list_dags [-h] [-sd SUBDIR] [-r]

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

-r, --report

Show DagBag loading report

Default: False

resetdb¶

Burn down and rebuild the metadata database

airflow resetdb [-h] [-y]

Named Arguments¶

-y, --yes

Do not prompt to confirm reset. Use with care!

Default: False

create_user¶

Create an admin account

airflow create_user [-h] [-r ROLE] [-u USERNAME] [-e EMAIL] [-f FIRSTNAME]
                    [-l LASTNAME] [-p PASSWORD] [--use_random_password]

Named Arguments¶

`-r, --role`	Role of the user. Existing roles include Admin, User, Op, Viewer, and Public
`-u, --username`	Username of the user
`-e, --email`	Email of the user
`-f, --firstname`
	First name of the user
`-l, --lastname`	Last name of the user
`-p, --password`	Password of the user
`--use_random_password`
	Do not prompt for password. Use random string instead Default: False

webserver¶

Start a Airflow webserver instance

airflow webserver [-h] [-p PORT] [-w WORKERS]
                  [-k {sync,eventlet,gevent,tornado}] [-t WORKER_TIMEOUT]
                  [-hn HOSTNAME] [--pid [PID]] [-D] [--stdout STDOUT]
                  [--stderr STDERR] [-A ACCESS_LOGFILE] [-E ERROR_LOGFILE]
                  [-l LOG_FILE] [--ssl_cert SSL_CERT] [--ssl_key SSL_KEY] [-d]

Named Arguments¶

`-p, --port`	The port on which to run the server Default: 8080
`-w, --workers`	Number of workers to run the webserver on Default: 4
`-k, --workerclass`
	Possible choices: sync, eventlet, gevent, tornado The worker class to use for Gunicorn Default: “sync”
`-t, --worker_timeout`
	The timeout for waiting on webserver workers Default: 120
`-hn, --hostname`
	Set the hostname on which to run the web server Default: “0.0.0.0”
`--pid`	PID file location
`-D, --daemon`	Daemonize instead of running in the foreground Default: False
`--stdout`	Redirect stdout to this file
`--stderr`	Redirect stderr to this file
`-A, --access_logfile`
	The logfile to store the webserver access log. Use ‘-‘ to print to stderr. Default: “-“
`-E, --error_logfile`
	The logfile to store the webserver error log. Use ‘-‘ to print to stderr. Default: “-“
`-l, --log-file`	Location of the log file
`--ssl_cert`	Path to the SSL certificate for the webserver
`--ssl_key`	Path to the key to use with the SSL certificate
`-d, --debug`	Use the server that ships with Flask in debug mode Default: False

pool¶

CRUD operations on pools

airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME] [-x NAME]

Named Arguments¶

`-s, --set`	Set pool slot count and description, respectively
`-g, --get`	Get pool info
`-x, --delete`	Delete a pool

scheduler¶

Start a scheduler instance

airflow scheduler [-h] [-d DAG_ID] [-sd SUBDIR] [-r RUN_DURATION]
                  [-n NUM_RUNS] [-p] [--pid [PID]] [-D] [--stdout STDOUT]
                  [--stderr STDERR] [-l LOG_FILE]

Named Arguments¶

`-d, --dag_id`	The id of the dag to run
`-sd, --subdir`	File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags”
`-r, --run-duration`
	Set number of seconds to execute before exiting
`-n, --num_runs`	Set the number of runs to execute before exiting Default: -1
`-p, --do_pickle`
	Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code. Default: False
`--pid`	PID file location
`-D, --daemon`	Daemonize instead of running in the foreground Default: False
`--stdout`	Redirect stdout to this file
`--stderr`	Redirect stderr to this file
`-l, --log-file`	Location of the log file

serve_logs¶

Serve logs generate by worker

airflow serve_logs [-h]

clear¶

Clear a set of task instance, as if they never ran

airflow clear [-h] [-t TASK_REGEX] [-s START_DATE] [-e END_DATE] [-sd SUBDIR]
              [-u] [-d] [-c] [-f] [-r] [-x] [-xp] [-dx]
              dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

`-t, --task_regex`
	The regex to filter specific task_ids to backfill (optional)
`-s, --start_date`
	Override start_date YYYY-MM-DD
`-e, --end_date`	Override end_date YYYY-MM-DD
`-sd, --subdir`	File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags”
`-u, --upstream`	Include upstream tasks Default: False
`-d, --downstream`
	Include downstream tasks Default: False
`-c, --no_confirm`
	Do not request confirmation Default: False
`-f, --only_failed`
	Only failed jobs Default: False
`-r, --only_running`
	Only running jobs Default: False
`-x, --exclude_subdags`
	Exclude subdags Default: False
`-xp, --exclude_parentdag`
	Exclude ParentDAGS if the task cleared is a part of a SubDAG Default: False
`-dx, --dag_regex`
	Search dag_id as regex instead of exact string Default: False

trigger_dag¶

Trigger a DAG run

airflow trigger_dag [-h] [-sd SUBDIR] [-r RUN_ID] [-c CONF] [-e EXEC_DATE]
                    dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

`-sd, --subdir`	File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags”
`-r, --run_id`	Helps to identify this run
`-c, --conf`	JSON string that gets pickled into the DagRun’s conf attribute
`-e, --exec_date`
	The execution date of the DAG

test¶

Test a task instance. This will run a task without checking for dependencies or recording its state in the database.

airflow test [-h] [-sd SUBDIR] [-dr] [-tp TASK_PARAMS]
             dag_id task_id execution_date

Positional Arguments¶

`dag_id`	The id of the dag
`task_id`	The id of the task
`execution_date`	The execution date of the DAG

Named Arguments¶

`-sd, --subdir`	File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags”
`-dr, --dry_run`	Perform a dry run Default: False
`-tp, --task_params`
	Sends a JSON params dict to the task

connections¶

List/Add/Delete connections

airflow connections [-h] [-l] [-a] [-d] [--conn_id CONN_ID]
                    [--conn_uri CONN_URI] [--conn_extra CONN_EXTRA]
                    [--conn_type CONN_TYPE] [--conn_host CONN_HOST]
                    [--conn_login CONN_LOGIN] [--conn_password CONN_PASSWORD]
                    [--conn_schema CONN_SCHEMA] [--conn_port CONN_PORT]

Named Arguments¶

`-l, --list`	List all connections Default: False
`-a, --add`	Add a connection Default: False
`-d, --delete`	Delete a connection Default: False
`--conn_id`	Connection id, required to add/delete a connection
`--conn_uri`	Connection URI, required to add a connection without conn_type
`--conn_extra`	Connection Extra field, optional when adding a connection
`--conn_type`	Connection type, required to add a connection without conn_uri
`--conn_host`	Connection host, optional when adding a connection
`--conn_login`	Connection login, optional when adding a connection
`--conn_password`
	Connection password, optional when adding a connection
`--conn_schema`	Connection schema, optional when adding a connection
`--conn_port`	Connection port, optional when adding a connection

worker¶

Start a Celery worker node

airflow worker [-h] [-p] [-q QUEUES] [-c CONCURRENCY] [-cn CELERY_HOSTNAME]
               [--pid [PID]] [-D] [--stdout STDOUT] [--stderr STDERR]
               [-l LOG_FILE]

Named Arguments¶

`-p, --do_pickle`
	Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code. Default: False
`-q, --queues`	Comma delimited list of queues to serve Default: “default”
`-c, --concurrency`
	The number of worker processes Default: 16
`-cn, --celery_hostname`
	Set the hostname of celery worker if you have multiple workers on a single machine.
`--pid`	PID file location
`-D, --daemon`	Daemonize instead of running in the foreground Default: False
`--stdout`	Redirect stdout to this file
`--stderr`	Redirect stderr to this file
`-l, --log-file`	Location of the log file

kerberos¶

Start a kerberos ticket renewer

airflow kerberos [-h] [-kt [KEYTAB]] [--pid [PID]] [-D] [--stdout STDOUT]
                 [--stderr STDERR] [-l LOG_FILE]
                 [principal]

Positional Arguments¶

principal

kerberos principal

Default: “airflow”

Named Arguments¶

`-kt, --keytab`	keytab Default: “airflow.keytab”
`--pid`	PID file location
`-D, --daemon`	Daemonize instead of running in the foreground Default: False
`--stdout`	Redirect stdout to this file
`--stderr`	Redirect stderr to this file
`-l, --log-file`	Location of the log file

pause¶

Pause a DAG

airflow pause [-h] [-sd SUBDIR] dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

task_failed_deps¶

Returns the unmet dependencies for a task instance from the perspective of the scheduler. In other words, why a task instance doesn’t get scheduled and then queued by the scheduler, and then run by an executor).

airflow task_failed_deps [-h] [-sd SUBDIR] dag_id task_id execution_date

Positional Arguments¶

`dag_id`	The id of the dag
`task_id`	The id of the task
`execution_date`	The execution date of the DAG

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

render¶

Render a task instance’s template(s)

airflow render [-h] [-sd SUBDIR] dag_id task_id execution_date

Positional Arguments¶

`dag_id`	The id of the dag
`task_id`	The id of the task
`execution_date`	The execution date of the DAG

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

run¶

Run a single task instance

airflow run [-h] [-sd SUBDIR] [-m] [-f] [--pool POOL] [--cfg_path CFG_PATH]
            [-l] [-A] [-i] [-I] [--ship_dag] [-p PICKLE] [-int]
            dag_id task_id execution_date

Positional Arguments¶

`dag_id`	The id of the dag
`task_id`	The id of the task
`execution_date`	The execution date of the DAG

Named Arguments¶

`-sd, --subdir`	File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags”
`-m, --mark_success`
	Mark jobs as succeeded without running them Default: False
`-f, --force`	Ignore previous task instance state, rerun regardless if task already succeeded/failed Default: False
`--pool`	Resource pool to use
`--cfg_path`	Path to config file to use instead of airflow.cfg
`-l, --local`	Run the task using the LocalExecutor Default: False
`-A, --ignore_all_dependencies`
	Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_deps Default: False
`-i, --ignore_dependencies`
	Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies Default: False
`-I, --ignore_depends_on_past`
	Ignore depends_on_past dependencies (but respect upstream dependencies) Default: False
`--ship_dag`	Pickles (serializes) the DAG and ships it to the worker Default: False
`-p, --pickle`	Serialized pickle object of the entire dag (used internally)
`-int, --interactive`
	Do not capture standard output and error streams (useful for interactive debugging) Default: False

list_tasks¶

List the tasks within a DAG

airflow list_tasks [-h] [-t] [-sd SUBDIR] dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

-t, --tree

Tree view

Default: False

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

backfill¶

Run subsections of a DAG for a specified date range. If reset_dag_run option is used, backfill will first prompt users whether airflow should clear all the previous dag_run and task_instances within the backfill date range. If rerun_failed_tasks is used, backfill will auto re-run the previous failed task instances within the backfill date range.

airflow backfill [-h] [-t TASK_REGEX] [-s START_DATE] [-e END_DATE] [-m] [-l]
                 [-x] [-i] [-I] [-sd SUBDIR] [--pool POOL]
                 [--delay_on_limit DELAY_ON_LIMIT] [-dr] [-v] [-c CONF]
                 [--reset_dagruns] [--rerun_failed_tasks]
                 dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

`-t, --task_regex`
	The regex to filter specific task_ids to backfill (optional)
`-s, --start_date`
	Override start_date YYYY-MM-DD
`-e, --end_date`	Override end_date YYYY-MM-DD
`-m, --mark_success`
	Mark jobs as succeeded without running them Default: False
`-l, --local`	Run the task using the LocalExecutor Default: False
`-x, --donot_pickle`
	Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code. Default: False
`-i, --ignore_dependencies`
	Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex Default: False
`-I, --ignore_first_depends_on_past`
	Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past). Default: False
`-sd, --subdir`	File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’ Default: “[AIRFLOW_HOME]/dags”
`--pool`	Resource pool to use
`--delay_on_limit`
	Amount of time in seconds to wait when the limit on maximum active dag runs (max_active_runs) has been reached before trying to execute a dag run again. Default: 1.0
`-dr, --dry_run`	Perform a dry run Default: False
`-v, --verbose`	Make logging output more verbose Default: False
`-c, --conf`	JSON string that gets pickled into the DagRun’s conf attribute
`--reset_dagruns`
	if set, the backfill will delete existing backfill-related DAG runs and start anew with fresh, running DAG runs Default: False
`--rerun_failed_tasks`
	if set, the backfill will auto-rerun all the failed tasks for the backfill date range instead of throwing exceptions Default: False

dag_state¶

Get the status of a dag run

airflow dag_state [-h] [-sd SUBDIR] dag_id execution_date

Positional Arguments¶

`dag_id`	The id of the dag
`execution_date`	The execution date of the DAG

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”

variables¶

CRUD operations on variables

airflow variables [-h] [-s KEY VAL] [-g KEY] [-j] [-d VAL] [-i FILEPATH]
                  [-e FILEPATH] [-x KEY]

Named Arguments¶

`-s, --set`	Set a variable
`-g, --get`	Get value of a variable
`-j, --json`	Deserialize JSON variable Default: False
`-d, --default`	Default value returned if variable does not exist
`-i, --import`	Import variables from JSON file
`-e, --export`	Export variables to JSON file
`-x, --delete`	Delete a variable

flower¶

Start a Celery Flower

airflow flower [-h] [-hn HOSTNAME] [-p PORT] [-fc FLOWER_CONF] [-u URL_PREFIX]
               [-a BROKER_API] [--pid [PID]] [-D] [--stdout STDOUT]
               [--stderr STDERR] [-l LOG_FILE]

Named Arguments¶

`-hn, --hostname`
	Set the hostname on which to run the server Default: “0.0.0.0”
`-p, --port`	The port on which to run the server Default: 5555
`-fc, --flower_conf`
	Configuration file for flower
`-u, --url_prefix`
	URL prefix for Flower
`-a, --broker_api`
	Broker api
`--pid`	PID file location
`-D, --daemon`	Daemonize instead of running in the foreground Default: False
`--stdout`	Redirect stdout to this file
`--stderr`	Redirect stderr to this file
`-l, --log-file`	Location of the log file

unpause¶

Resume a paused DAG

airflow unpause [-h] [-sd SUBDIR] dag_id

Positional Arguments¶

dag_id

The id of the dag

Named Arguments¶

-sd, --subdir

File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’

Default: “[AIRFLOW_HOME]/dags”