Configuration Reference¶

This page contains the list of all the available Airflow configurations that you can set in airflow.cfg file or using environment variables.

Use the same configuration across all the Airflow components. While each component does not require all, some configurations need to be same otherwise they would not work as expected. A good example for that is secret_key which should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker.

The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token generated using the secret key has a short expiry time though - make sure that time on ALL the machines that you run airflow components on is synchronized (for example using ntpd) otherwise you might get “forbidden” errors when the logs are accessed.

Note

For more information see Setting Configuration Options.

Provider-specific configuration options¶

Some of the providers have their own configuration options, you will find details of their configuration in the provider’s documentation.

You can find all the provider configuration in configurations specific for provider packages

Airflow configuration options¶

Sections:

[api]
[cli]
[core]
[database]
[debug]
[email]
[kerberos]
[lineage]
[logging]
[metrics]
[operators]
[scheduler]
[secrets]
[sensors]
[sentry]
[smtp]
[triggerer]
[webserver]

[api]¶

access_control_allow_headers¶

New in version 2.1.0.

Used in response to a preflight request to indicate which HTTP headers can be used when making the actual request. This header is the server side response to the browser’s Access-Control-Request-Headers header.

Type: string
Default: ''
Environment Variable: AIRFLOW__API__ACCESS_CONTROL_ALLOW_HEADERS

access_control_allow_methods¶

New in version 2.1.0.

Specifies the method or methods allowed when accessing the resource.

Type: string
Default: ''
Environment Variable: AIRFLOW__API__ACCESS_CONTROL_ALLOW_METHODS

access_control_allow_origins¶

New in version 2.2.0.

Indicates whether the response can be shared with requesting code from the given origins. Separate URLs with space.

Type: string
Default: ''
Environment Variable: AIRFLOW__API__ACCESS_CONTROL_ALLOW_ORIGINS

auth_backends¶

New in version 2.3.0.

Comma separated list of auth backends to authenticate users of the API. See Security: API for possible values. (“airflow.api.auth.backend.default” allows all requests for historic reasons)

Type: string
Default: airflow.api.auth.backend.session
Environment Variable: AIRFLOW__API__AUTH_BACKENDS

enable_experimental_api¶

New in version 2.0.0.

Enables the deprecated experimental API. Please note that these APIs do not have access control. The authenticated user has full access.

Warning

This Experimental REST API is deprecated since version 2.0. Please consider using the Stable REST API. For more information on migration, see RELEASE_NOTES.rst

Type: boolean
Default: False
Environment Variable: AIRFLOW__API__ENABLE_EXPERIMENTAL_API

enable_xcom_deserialize_support¶

New in version 2.7.0.

Indicates whether the xcomEntries endpoint supports the deserialize flag. If set to False, setting this flag in a request would result in a 400 Bad Request error.

Type: boolean
Default: False
Environment Variable: AIRFLOW__API__ENABLE_XCOM_DESERIALIZE_SUPPORT

fallback_page_limit¶

New in version 2.0.0.

Used to set the default page limit when limit param is zero or not provided in API requests. Otherwise if positive integer is passed in the API requests as limit, the smallest number of user given limit or maximum page limit is taken as limit.

Type: integer
Default: 100
Environment Variable: AIRFLOW__API__FALLBACK_PAGE_LIMIT

google_key_path¶

New in version 2.0.0.

Path to Google Cloud Service Account key file (JSON). If omitted, authorization based on the Application Default Credentials will be used.

Type: string
Default: ''
Environment Variable: AIRFLOW__API__GOOGLE_KEY_PATH
Example: /files/service-account-json

google_oauth2_audience¶

New in version 2.0.0.

The intended audience for JWT token credentials used for authorization. This value must match on the client and server sides. If empty, audience will not be tested.

Type: string
Default: ''
Environment Variable: AIRFLOW__API__GOOGLE_OAUTH2_AUDIENCE
Example: project-id-random-value.apps.googleusercontent.com

maximum_page_limit¶

New in version 2.0.0.

Used to set the maximum page limit for API requests. If limit passed as param is greater than maximum page limit, it will be ignored and maximum page limit value will be set as the limit

Type: integer
Default: 100
Environment Variable: AIRFLOW__API__MAXIMUM_PAGE_LIMIT

access_control_allow_origin (Deprecated)¶

Deprecated since version 2.2.0: The option has been moved to api.access_control_allow_origins

auth_backend (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to api.auth_backends

[cli]¶

api_client¶

In what way should the cli access the API. The LocalClient will use the database directly, while the json_client will use the api running on the webserver

Type: string
Default: airflow.api.client.local_client
Environment Variable: AIRFLOW__CLI__API_CLIENT

endpoint_url¶

If you set web_server_url_prefix, do NOT forget to append it here, ex: endpoint_url = http://localhost:8080/myroot So api will look like: http://localhost:8080/myroot/api/experimental/...

Type: string
Default: http://localhost:8080
Environment Variable: AIRFLOW__CLI__ENDPOINT_URL

[core]¶

allowed_deserialization_classes¶

New in version 2.5.0.

What classes can be imported during deserialization. This is a multi line value. The individual items will be parsed as a pattern to a glob function. Python built-in classes (like dict) are always allowed.

Type: string
Default: airflow.*
Environment Variable: AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES

allowed_deserialization_classes_regexp¶

New in version 2.8.2.

What classes can be imported during deserialization. This is a multi line value. The individual items will be parsed as regexp patterns. This is a secondary option to [core] allowed_deserialization_classes.

Type: string
Default: ''
Environment Variable: AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES_REGEXP

auth_manager¶

New in version 2.7.0.

The auth manager class that airflow should use. Full import path to the auth manager class.

Type: string
Default: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
Environment Variable: AIRFLOW__CORE__AUTH_MANAGER

check_slas¶

New in version 1.10.8.

On each dagrun check against defined SLAs

Type: string
Default: True
Environment Variable: AIRFLOW__CORE__CHECK_SLAS

compress_serialized_dags¶

New in version 2.3.0.

If True, serialized DAGs are compressed before writing to DB.

Note

This will disable the DAG dependencies view

Type: string
Default: False
Environment Variable: AIRFLOW__CORE__COMPRESS_SERIALIZED_DAGS

daemon_umask¶

New in version 2.3.4.

The default umask to use for process when run in daemon mode (scheduler, worker, etc.)

This controls the file-creation mode mask which determines the initial value of file permission bits for newly created files.

This value is treated as an octal-integer.

Type: string
Default: 0o077
Environment Variable: AIRFLOW__CORE__DAEMON_UMASK

dag_discovery_safe_mode¶

New in version 1.10.3.

If enabled, Airflow will only scan files containing both DAG and airflow (case-insensitive).

Type: string
Default: True
Environment Variable: AIRFLOW__CORE__DAG_DISCOVERY_SAFE_MODE

dag_file_processor_timeout¶

New in version 1.10.6.

How long before timing out a DagFileProcessor, which processes a dag file

Type: string
Default: 50
Environment Variable: AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT

dag_ignore_file_syntax¶

New in version 2.3.0.

The pattern syntax used in the .airflowignore files in the DAG directories. Valid values are regexp or glob.

Type: string
Default: regexp
Environment Variable: AIRFLOW__CORE__DAG_IGNORE_FILE_SYNTAX

dag_run_conf_overrides_params¶

Whether to override params with dag_run.conf. If you pass some key-value pairs through airflow dags backfill -c or airflow dags trigger -c, the key-value pairs will override the existing ones in params.

Type: string
Default: True
Environment Variable: AIRFLOW__CORE__DAG_RUN_CONF_OVERRIDES_PARAMS

dagbag_import_error_traceback_depth¶

New in version 2.0.0.

If tracebacks are shown, how many entries from the traceback should be shown

Type: integer
Default: 2
Environment Variable: AIRFLOW__CORE__DAGBAG_IMPORT_ERROR_TRACEBACK_DEPTH

dagbag_import_error_tracebacks¶

New in version 2.0.0.

Should a traceback be shown in the UI for dagbag import errors, instead of just the exception message

Type: boolean
Default: True
Environment Variable: AIRFLOW__CORE__DAGBAG_IMPORT_ERROR_TRACEBACKS

dagbag_import_timeout¶

How long before timing out a python file import

Type: float
Default: 30.0
Environment Variable: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT

dags_are_paused_at_creation¶

Are DAGs paused by default at creation

Type: string
Default: True
Environment Variable: AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION

dags_folder¶

The folder where your airflow pipelines live, most likely a subfolder in a code repository. This path must be absolute.

Type: string
Default: {AIRFLOW_HOME}/dags
Environment Variable: AIRFLOW__CORE__DAGS_FOLDER

database_access_isolation¶

New in version 2.6.0.

(experimental) Whether components should use Airflow Internal API for DB connectivity.

Type: boolean
Default: False
Environment Variable: AIRFLOW__CORE__DATABASE_ACCESS_ISOLATION

dataset_manager_class¶

New in version 2.4.0.

Class to use as dataset manager.

Type: string
Default: None
Environment Variable: AIRFLOW__CORE__DATASET_MANAGER_CLASS
Example: airflow.datasets.manager.DatasetManager

dataset_manager_kwargs¶

New in version 2.4.0.

Kwargs to supply to dataset manager.

Type

string

Default

None

Environment Variables

AIRFLOW__CORE__DATASET_MANAGER_KWARGS

AIRFLOW__CORE__DATASET_MANAGER_KWARGS_CMD

AIRFLOW__CORE__DATASET_MANAGER_KWARGS_SECRET

Example

{"some_param": "some_value"}

default_impersonation¶

If set, tasks without a run_as_user argument will be run with this user Can be used to de-elevate a sudo user running Airflow when executing tasks

Type: string
Default: ''
Environment Variable: AIRFLOW__CORE__DEFAULT_IMPERSONATION

default_pool_task_slot_count¶

New in version 2.2.0.

Task Slot counts for default_pool. This setting would not have any effect in an existing deployment where the default_pool is already created. For existing deployments, users can change the number of slots using Webserver, API or the CLI

Type: string
Default: 128
Environment Variable: AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT

default_task_execution_timeout¶

New in version 2.3.0.

The default task execution_timeout value for the operators. Expected an integer value to be passed into timedelta as seconds. If not specified, then the value is considered as None, meaning that the operators are never timed out by default.

Type: integer
Default: ''
Environment Variable: AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT

default_task_retries¶

New in version 1.10.6.

The number of retries each task is going to have by default. Can be overridden at dag or task level.

Type: string
Default: 0
Environment Variable: AIRFLOW__CORE__DEFAULT_TASK_RETRIES

default_task_retry_delay¶

New in version 2.4.0.

The number of seconds each task is going to wait by default between retries. Can be overridden at dag or task level.

Type: integer
Default: 300
Environment Variable: AIRFLOW__CORE__DEFAULT_TASK_RETRY_DELAY

default_task_weight_rule¶

New in version 2.2.0.

The weighting method used for the effective total priority weight of the task

Type: string
Default: downstream
Environment Variable: AIRFLOW__CORE__DEFAULT_TASK_WEIGHT_RULE

default_timezone¶

Default timezone in case supplied date times are naive can be UTC (default), system, or any IANA <https://www.iana.org/time-zones> timezone string (e.g. Europe/Amsterdam)

Type: string
Default: utc
Environment Variable: AIRFLOW__CORE__DEFAULT_TIMEZONE

donot_pickle¶

Whether to disable pickling dags

Type: string
Default: True
Environment Variable: AIRFLOW__CORE__DONOT_PICKLE

enable_xcom_pickling¶

Whether to enable pickling for xcom (note that this is insecure and allows for RCE exploits).

Type: string
Default: False
Environment Variable: AIRFLOW__CORE__ENABLE_XCOM_PICKLING

execute_tasks_new_python_interpreter¶

New in version 2.0.0.

Should tasks be executed via forking of the parent process

False: Execute via forking of the parent process
True: Spawning a new python process, slower than fork, but means plugin changes picked up by tasks straight away

See also

When are plugins (re)loaded?

Type: boolean
Default: False
Environment Variable: AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER

executor¶

The executor class that airflow should use. Choices include SequentialExecutor, LocalExecutor, CeleryExecutor, KubernetesExecutor, CeleryKubernetesExecutor, LocalKubernetesExecutor or the full import path to the class when using a custom executor.

Type: string
Default: SequentialExecutor
Environment Variable: AIRFLOW__CORE__EXECUTOR

fernet_key¶

Secret key to save connection passwords in the db

Type

string

Default

{FERNET_KEY}

Environment Variables

AIRFLOW__CORE__FERNET_KEY

AIRFLOW__CORE__FERNET_KEY_CMD

AIRFLOW__CORE__FERNET_KEY_SECRET

hide_sensitive_var_conn_fields¶

New in version 2.1.0.

Hide sensitive Variables or Connection extra json keys from UI and task logs when set to True

Note

Connection passwords are always hidden in logs

Type: boolean
Default: True
Environment Variable: AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS

hostname_callable¶

Hostname by providing a path to a callable, which will resolve the hostname. The format is “package.function”.

For example, default value airflow.utils.net.getfqdn means that result from patched version of socket.getfqdn(), see related CPython Issue.

No argument should be required in the function specified. If using IP address as hostname is preferred, use value airflow.utils.net.get_host_ip_address

Type: string
Default: airflow.utils.net.getfqdn
Environment Variable: AIRFLOW__CORE__HOSTNAME_CALLABLE

internal_api_url¶

New in version 2.6.0.

(experimental) Airflow Internal API url. Only used if [core] database_access_isolation is True.

Type: string
Default: None
Environment Variable: AIRFLOW__CORE__INTERNAL_API_URL
Example: http://localhost:8080

killed_task_cleanup_time¶

When a task is killed forcefully, this is the amount of time in seconds that it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED

Type: string
Default: 60
Environment Variable: AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME

lazy_discover_providers¶

New in version 2.0.0.

By default Airflow providers are lazily-discovered (discovery and imports happen only when required). Set it to False, if you want to discover providers whenever ‘airflow’ is invoked via cli or loaded from module.

Type: boolean
Default: True
Environment Variable: AIRFLOW__CORE__LAZY_DISCOVER_PROVIDERS

lazy_load_plugins¶

New in version 2.0.0.

By default Airflow plugins are lazily-loaded (only loaded when required). Set it to False, if you want to load plugins whenever ‘airflow’ is invoked via cli or loaded from module.

Type: boolean
Default: True
Environment Variable: AIRFLOW__CORE__LAZY_LOAD_PLUGINS

load_examples¶

Whether to load the DAG examples that ship with Airflow. It’s good to get started, but you probably want to set this to False in a production environment

Type: string
Default: True
Environment Variable: AIRFLOW__CORE__LOAD_EXAMPLES

max_active_runs_per_dag¶

The maximum number of active DAG runs per DAG. The scheduler will not create more DAG runs if it reaches the limit. This is configurable at the DAG level with max_active_runs, which is defaulted as [core] max_active_runs_per_dag.

Type: string
Default: 16
Environment Variable: AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG

max_active_tasks_per_dag¶

New in version 2.2.0.

The maximum number of task instances allowed to run concurrently in each DAG. To calculate the number of tasks that is running concurrently for a DAG, add up the number of running tasks for all DAG runs of the DAG. This is configurable at the DAG level with max_active_tasks, which is defaulted as [core] max_active_tasks_per_dag.

An example scenario when this would be useful is when you want to stop a new dag with an early start date from stealing all the executor slots in a cluster.

Type: string
Default: 16
Environment Variable: AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG

max_consecutive_failed_dag_runs_per_dag¶

New in version 2.9.0.

(experimental) The maximum number of consecutive DAG failures before DAG is automatically paused. This is also configurable per DAG level with max_consecutive_failed_dag_runs, which is defaulted as [core] max_consecutive_failed_dag_runs_per_dag. If not specified, then the value is considered as 0, meaning that the dags are never paused out by default.

Type: string
Default: 0
Environment Variable: AIRFLOW__CORE__MAX_CONSECUTIVE_FAILED_DAG_RUNS_PER_DAG

max_map_length¶

New in version 2.3.0.

The maximum list/dict length an XCom can push to trigger task mapping. If the pushed list/dict has a length exceeding this value, the task pushing the XCom will be failed automatically to prevent the mapped tasks from clogging the scheduler.

Type: integer
Default: 1024
Environment Variable: AIRFLOW__CORE__MAX_MAP_LENGTH

max_num_rendered_ti_fields_per_task¶

New in version 1.10.10.

Maximum number of Rendered Task Instance Fields (Template Fields) per task to store in the Database. All the template_fields for each of Task Instance are stored in the Database. Keeping this number small may cause an error when you try to view Rendered tab in TaskInstance view for older tasks.

Type: integer
Default: 30
Environment Variable: AIRFLOW__CORE__MAX_NUM_RENDERED_TI_FIELDS_PER_TASK

max_task_retry_delay¶

New in version 2.6.0.

The maximum delay (in seconds) each task is going to wait by default between retries. This is a global setting and cannot be overridden at task or DAG level.

Type: integer
Default: 86400
Environment Variable: AIRFLOW__CORE__MAX_TASK_RETRY_DELAY

max_templated_field_length¶

New in version 2.9.0.

The maximum length of the rendered template field. If the value to be stored in the rendered template field exceeds this size, it’s redacted.

Type: integer
Default: 4096
Environment Variable: AIRFLOW__CORE__MAX_TEMPLATED_FIELD_LENGTH

might_contain_dag_callable¶

New in version 2.6.0.

A callable to check if a python file has airflow dags defined or not and should return True if it has dags otherwise False. If this is not provided, Airflow uses its own heuristic rules.

The function should have the following signature

def func_name(file_path: str, zip_file: zipfile.ZipFile | None = None) -> bool: ...

Type: string
Default: airflow.utils.file.might_contain_dag_via_default_heuristic
Environment Variable: AIRFLOW__CORE__MIGHT_CONTAIN_DAG_CALLABLE

min_serialized_dag_fetch_interval¶

New in version 1.10.12.

Fetching serialized DAG can not be faster than a minimum interval to reduce database read rate. This config controls when your DAGs are updated in the Webserver

Type: string
Default: 10
Environment Variable: AIRFLOW__CORE__MIN_SERIALIZED_DAG_FETCH_INTERVAL

min_serialized_dag_update_interval¶

New in version 1.10.7.

Updating serialized DAG can not be faster than a minimum interval to reduce database write rate.

Type: string
Default: 30
Environment Variable: AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL

mp_start_method¶

New in version 2.0.0.

The name of the method used in order to start Python processes via the multiprocessing module. This corresponds directly with the options available in the Python docs: multiprocessing.set_start_method must be one of the values returned by multiprocessing.get_all_start_methods().

Type: string
Default: None
Environment Variable: AIRFLOW__CORE__MP_START_METHOD
Example: fork

parallelism¶

This defines the maximum number of task instances that can run concurrently per scheduler in Airflow, regardless of the worker count. Generally this value, multiplied by the number of schedulers in your cluster, is the maximum number of task instances with the running state in the metadata database.

Type: string
Default: 32
Environment Variable: AIRFLOW__CORE__PARALLELISM

plugins_folder¶

Path to the folder containing Airflow plugins

Type: string
Default: {AIRFLOW_HOME}/plugins
Environment Variable: AIRFLOW__CORE__PLUGINS_FOLDER

security¶

What security module to use (for example kerberos)

Type: string
Default: ''
Environment Variable: AIRFLOW__CORE__SECURITY

sensitive_var_conn_names¶

New in version 2.1.0.

A comma-separated list of extra sensitive keywords to look for in variables names or connection’s extra JSON.

Type: string
Default: ''
Environment Variable: AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES

strict_dataset_uri_validation¶

New in version 2.9.2.

Dataset URI validation should raise an exception if it is not compliant with AIP-60. By default this configuration is false, meaning that Airflow 2.x only warns the user. In Airflow 3, this configuration will be enabled by default.

Type: boolean
Default: False
Environment Variable: AIRFLOW__CORE__STRICT_DATASET_URI_VALIDATION

task_runner¶

The class to use for running task instances in a subprocess. Choices include StandardTaskRunner, CgroupTaskRunner or the full import path to the class when using a custom task runner.

Type: string
Default: StandardTaskRunner
Environment Variable: AIRFLOW__CORE__TASK_RUNNER

test_connection¶

New in version 2.7.0.

The ability to allow testing connections across Airflow UI, API and CLI. Supported options: Disabled, Enabled, Hidden. Default: Disabled Disabled - Disables the test connection functionality and disables the Test Connection button in UI. Enabled - Enables the test connection functionality and shows the Test Connection button in UI. Hidden - Disables the test connection functionality and hides the Test Connection button in UI. Before setting this to Enabled, make sure that you review the users who are able to add/edit connections and ensure they are trusted. Connection testing can be done maliciously leading to undesired and insecure outcomes. See Airflow Security Model: Capabilities of authenticated UI users for more details.

Type: string
Default: Disabled
Environment Variable: AIRFLOW__CORE__TEST_CONNECTION

unit_test_mode¶

Turn unit test mode on (overwrites many configuration options with test values at runtime)

Type: string
Default: False
Environment Variable: AIRFLOW__CORE__UNIT_TEST_MODE

xcom_backend¶

New in version 1.10.12.

Path to custom XCom class that will be used to store and resolve operators results

Type: string
Default: airflow.models.xcom.BaseXCom
Environment Variable: AIRFLOW__CORE__XCOM_BACKEND
Example: path.to.CustomXCom

base_log_folder (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.base_log_folder

colored_console_log (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.colored_console_log

colored_formatter_class (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.colored_formatter_class

colored_log_format (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.colored_log_format

dag_concurrency (Deprecated)¶

Deprecated since version 2.2.0: The option has been moved to core.max_active_tasks_per_dag

dag_processor_manager_log_location (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.dag_processor_manager_log_location

encrypt_s3_logs (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.encrypt_s3_logs

fab_logging_level (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.fab_logging_level

interleave_timestamp_parser (Deprecated)¶

Deprecated since version 2.6.1: The option has been moved to logging.interleave_timestamp_parser

load_default_connections (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.load_default_connections

log_filename_template (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.log_filename_template

log_format (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.log_format

log_processor_filename_template (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.log_processor_filename_template

logging_config_class (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.logging_config_class

logging_level (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.logging_level

max_db_retries (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.max_db_retries

non_pooled_task_slot_count (Deprecated)¶

Deprecated since version 1.10.4: The option has been moved to core.default_pool_task_slot_count

remote_base_log_folder (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.remote_base_log_folder

remote_log_conn_id (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.remote_log_conn_id

remote_logging (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.remote_logging

simple_log_format (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.simple_log_format

sql_alchemy_conn (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_conn

sql_alchemy_connect_args (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_connect_args

sql_alchemy_max_overflow (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_max_overflow

sql_alchemy_pool_enabled (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_pool_enabled

sql_alchemy_pool_pre_ping (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_pool_pre_ping

sql_alchemy_pool_recycle (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_pool_recycle

sql_alchemy_pool_size (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_pool_size

sql_alchemy_schema (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_alchemy_schema

sql_engine_collation_for_ids (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_engine_collation_for_ids

sql_engine_encoding (Deprecated)¶

Deprecated since version 2.3.0: The option has been moved to database.sql_engine_encoding

task_log_prefix_template (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.task_log_prefix_template

task_log_reader (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to logging.task_log_reader

worker_precheck (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to celery.worker_precheck

[database]¶

alembic_ini_file_path¶

New in version 2.7.0.

Path to the alembic.ini file. You can either provide the file path relative to the Airflow home directory or the absolute path if it is located elsewhere.

Type: string
Default: alembic.ini
Environment Variable: AIRFLOW__DATABASE__ALEMBIC_INI_FILE_PATH

check_migrations¶

New in version 2.6.0.

Whether to run alembic migrations during Airflow start up. Sometimes this operation can be expensive, and the users can assert the correct version through other means (e.g. through a Helm chart). Accepts True or False.

Type: string
Default: True
Environment Variable: AIRFLOW__DATABASE__CHECK_MIGRATIONS

load_default_connections¶

New in version 2.3.0.

Whether to load the default connections that ship with Airflow when airflow db init is called. It’s good to get started, but you probably want to set this to False in a production environment.

Deprecated since version 2.7.0: This option is only used by the deprecated “airflow db init” command. This option has been used in previous versions of Airflow to determine if loading of the default connections is done with the airflow db init command. This command has been deprecated and replaced by two separate commands airflow db migrate and airflow connections create-default-connections and load_default_connections is not used anymore by those commands.

Type: string
Default: True
Environment Variable: AIRFLOW__DATABASE__LOAD_DEFAULT_CONNECTIONS

max_db_retries¶

New in version 2.3.0.

Number of times the code should be retried in case of DB Operational Errors. Not all transactions will be retried as it can cause undesired state. Currently it is only used in DagFileProcessor.process_file to retry dagbag.sync_to_db.

Type: integer
Default: 3
Environment Variable: AIRFLOW__DATABASE__MAX_DB_RETRIES

sql_alchemy_conn¶

New in version 2.3.0.

The SQLAlchemy connection string to the metadata database. SQLAlchemy supports many different database engines. See: Set up a Database Backend: Database URI for more details.

Type

string

Default

sqlite:///{AIRFLOW_HOME}/airflow.db

Environment Variables

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_SECRET

sql_alchemy_connect_args¶

New in version 2.3.0.

Import path for connect args in SQLAlchemy. Defaults to an empty dict. This is useful when you want to configure db engine args that SQLAlchemy won’t parse in connection string. This can be set by passing a dictionary containing the create engine parameters. For more details about passing create engine parameters (keepalives variables, timeout etc) in Postgres DB Backend see Setting up a PostgreSQL Database e.g connect_args={"timeout":30} can be defined in airflow_local_settings.py and can be imported as shown below

Type: string
Default: None
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_CONNECT_ARGS
Example: airflow_local_settings.connect_args

sql_alchemy_engine_args¶

New in version 2.3.0.

Extra engine specific keyword args passed to SQLAlchemy’s create_engine, as a JSON-encoded value

Type

string

Default

None

Environment Variables

AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS

AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS_CMD

AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS_SECRET

Example

{"arg1": true}

sql_alchemy_max_overflow¶

New in version 2.3.0.

The maximum overflow size of the pool. When the number of checked-out connections reaches the size set in pool_size, additional connections will be returned up to this limit. When those additional connections are returned to the pool, they are disconnected and discarded. It follows then that the total number of simultaneous connections the pool will allow is pool_size + max_overflow, and the total number of “sleeping” connections the pool will allow is pool_size. max_overflow can be set to -1 to indicate no overflow limit; no limit will be placed on the total number of concurrent connections. Defaults to 10.

Type: string
Default: 10
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW

sql_alchemy_pool_enabled¶

New in version 2.3.0.

If SQLAlchemy should pool database connections.

Type: string
Default: True
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED

sql_alchemy_pool_pre_ping¶

New in version 2.3.0.

Check connection at the start of each connection pool checkout. Typically, this is a simple statement like “SELECT 1”. See SQLAlchemy Pooling: Disconnect Handling - Pessimistic for more details.

Type: string
Default: True
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING

sql_alchemy_pool_recycle¶

New in version 2.3.0.

The SQLAlchemy pool recycle is the number of seconds a connection can be idle in the pool before it is invalidated. This config does not apply to sqlite. If the number of DB connections is ever exceeded, a lower config value will allow the system to recover faster.

Type: string
Default: 1800
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE

sql_alchemy_pool_size¶

New in version 2.3.0.

The SQLAlchemy pool size is the maximum number of database connections in the pool. 0 indicates no limit.

Type: string
Default: 5
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE

sql_alchemy_schema¶

New in version 2.3.0.

The schema to use for the metadata database. SQLAlchemy supports databases with the concept of multiple schemas.

Type: string
Default: ''
Environment Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_SCHEMA

sql_engine_collation_for_ids¶

New in version 2.3.0.

Collation for dag_id, task_id, key, external_executor_id columns in case they have different encoding. By default this collation is the same as the database collation, however for mysql and mariadb the default is utf8mb3_bin so that the index sizes of our index keys will not exceed the maximum size of allowed index when collation is set to utf8mb4 variant, see GitHub Issue Comment for more details.

Type: string
Default: None
Environment Variable: AIRFLOW__DATABASE__SQL_ENGINE_COLLATION_FOR_IDS

sql_engine_encoding¶

New in version 2.3.0.

The encoding for the databases

Type: string
Default: utf-8
Environment Variable: AIRFLOW__DATABASE__SQL_ENGINE_ENCODING

[debug]¶

fail_fast¶

New in version 1.10.8.

Used only with DebugExecutor. If set to True DAG will fail with first failed task. Helpful for debugging purposes.

Type: string
Default: False
Environment Variable: AIRFLOW__DEBUG__FAIL_FAST

[email]¶

Configuration email backend and whether to send email alerts on retry or failure

default_email_on_failure¶

New in version 2.0.0.

Whether email alerts should be sent when a task failed

Type: boolean
Default: True
Environment Variable: AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_FAILURE

default_email_on_retry¶

New in version 2.0.0.

Whether email alerts should be sent when a task is retried

Type: boolean
Default: True
Environment Variable: AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_RETRY

email_backend¶

Email backend to use

Type: string
Default: airflow.utils.email.send_email_smtp
Environment Variable: AIRFLOW__EMAIL__EMAIL_BACKEND

email_conn_id¶

New in version 2.1.0.

Email connection to use

Type: string
Default: smtp_default
Environment Variable: AIRFLOW__EMAIL__EMAIL_CONN_ID

from_email¶

New in version 2.2.4.

Email address that will be used as sender address. It can either be raw email or the complete address in a format Sender Name <sender@email.com>

Type: string
Default: None
Environment Variable: AIRFLOW__EMAIL__FROM_EMAIL
Example: Airflow <airflow@example.com>

html_content_template¶

New in version 2.0.1.

File that will be used as the template for Email content (which will be rendered using Jinja2). If not set, Airflow uses a base template.

See also

Email Configuration

Type: string
Default: None
Environment Variable: AIRFLOW__EMAIL__HTML_CONTENT_TEMPLATE
Example: /path/to/my_html_content_template_file

ssl_context¶

New in version 2.7.0.

ssl context to use when using SMTP and IMAP SSL connections. By default, the context is “default” which sets it to ssl.create_default_context() which provides the right balance between compatibility and security, it however requires that certificates in your operating system are updated and that SMTP/IMAP servers of yours have valid certificates that have corresponding public keys installed on your machines. You can switch it to “none” if you want to disable checking of the certificates, but it is not recommended as it allows MITM (man-in-the-middle) attacks if your infrastructure is not sufficiently secured. It should only be set temporarily while you are fixing your certificate configuration. This can be typically done by upgrading to newer version of the operating system you run Airflow components on,by upgrading/refreshing proper certificates in the OS or by updating certificates for your mail servers.

Type: string
Default: default
Environment Variable: AIRFLOW__EMAIL__SSL_CONTEXT
Example: default

subject_template¶

New in version 2.0.1.

File that will be used as the template for Email subject (which will be rendered using Jinja2). If not set, Airflow uses a base template.

See also

Email Configuration

Type: string
Default: None
Environment Variable: AIRFLOW__EMAIL__SUBJECT_TEMPLATE
Example: /path/to/my_subject_template_file

[kerberos]¶

ccache¶

Location of your ccache file once kinit has been performed.

Type: string
Default: /tmp/airflow_krb5_ccache
Environment Variable: AIRFLOW__KERBEROS__CCACHE

forwardable¶

New in version 2.2.0.

Allow to disable ticket forwardability.

Type: boolean
Default: True
Environment Variable: AIRFLOW__KERBEROS__FORWARDABLE

include_ip¶

New in version 2.2.0.

Allow to remove source IP from token, useful when using token behind NATted Docker host.

Type: boolean
Default: True
Environment Variable: AIRFLOW__KERBEROS__INCLUDE_IP

keytab¶

Designates the path to the Kerberos keytab file for the Airflow user

Type: string
Default: airflow.keytab
Environment Variable: AIRFLOW__KERBEROS__KEYTAB

kinit_path¶

Path to the kinit executable

Type: string
Default: kinit
Environment Variable: AIRFLOW__KERBEROS__KINIT_PATH

principal¶

gets augmented with fqdn

Type: string
Default: airflow
Environment Variable: AIRFLOW__KERBEROS__PRINCIPAL

reinit_frequency¶

Determines the frequency at which initialization or re-initialization processes occur.

Type: string
Default: 3600
Environment Variable: AIRFLOW__KERBEROS__REINIT_FREQUENCY

[lineage]¶

backend¶

what lineage backend to use

Type: string
Default: ''
Environment Variable: AIRFLOW__LINEAGE__BACKEND

[logging]¶

base_log_folder¶

New in version 2.0.0.

The folder where airflow should store its log files. This path must be absolute. There are a few existing configurations that assume this is set to the default. If you choose to override this you may need to update the [logging] dag_processor_manager_log_location and [logging] child_process_log_directory settings as well.

Type: string
Default: {AIRFLOW_HOME}/logs
Environment Variable: AIRFLOW__LOGGING__BASE_LOG_FOLDER

celery_logging_level¶

New in version 2.3.0.

Logging level for celery. If not set, it uses the value of logging_level

Supported values: CRITICAL, ERROR, WARNING, INFO, DEBUG.

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__CELERY_LOGGING_LEVEL

celery_stdout_stderr_separation¶

New in version 2.7.0.

By default Celery sends all logs into stderr. If enabled any previous logging handlers will get removed. With this option AirFlow will create new handlers and send low level logs like INFO and WARNING to stdout, while sending higher severity logs to stderr.

Type: boolean
Default: False
Environment Variable: AIRFLOW__LOGGING__CELERY_STDOUT_STDERR_SEPARATION

colored_console_log¶

New in version 2.0.0.

Flag to enable/disable Colored logs in Console Colour the logs when the controlling terminal is a TTY.

Type: string
Default: True
Environment Variable: AIRFLOW__LOGGING__COLORED_CONSOLE_LOG

colored_formatter_class¶

New in version 2.0.0.

Specifies the class utilized by Airflow to implement colored logging

Type: string
Default: airflow.utils.log.colored_log.CustomTTYColoredFormatter
Environment Variable: AIRFLOW__LOGGING__COLORED_FORMATTER_CLASS

colored_log_format¶

New in version 2.0.0.

Log format for when Colored logs is enabled

Type: string
Default: [%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
Environment Variable: AIRFLOW__LOGGING__COLORED_LOG_FORMAT

dag_processor_log_format¶

New in version 2.4.0.

Format of Dag Processor Log line

Type: string
Default: [%%(asctime)s] [SOURCE:DAG_PROCESSOR] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
Environment Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_FORMAT

dag_processor_log_target¶

New in version 2.4.0.

Where to send dag parser logs. If “file”, logs are sent to log files defined by child_process_log_directory.

Type: string
Default: file
Environment Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET

dag_processor_manager_log_location¶

New in version 2.0.0.

Full path of dag_processor_manager logfile.

Type: string
Default: {AIRFLOW_HOME}/logs/dag_processor_manager/dag_processor_manager.log
Environment Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION

dag_processor_manager_log_stdout¶

New in version 2.9.0.

Whether DAG processor manager will write logs to stdout

Type: boolean
Default: False
Environment Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_STDOUT

delete_local_logs¶

New in version 2.6.0.

Whether the local log files for GCS, S3, WASB and OSS remote logging should be deleted after they are uploaded to the remote location.

Type: string
Default: False
Environment Variable: AIRFLOW__LOGGING__DELETE_LOCAL_LOGS

enable_task_context_logger¶

New in version 2.8.0.

If enabled, Airflow may ship messages to task logs from outside the task run context, e.g. from the scheduler, executor, or callback execution context. This can help in circumstances such as when there’s something blocking the execution of the task and ordinarily there may be no task logs at all. This is set to True by default. If you encounter issues with this feature (e.g. scheduler performance issues) it can be disabled.

Type: boolean
Default: True
Environment Variable: AIRFLOW__LOGGING__ENABLE_TASK_CONTEXT_LOGGER

encrypt_s3_logs¶

New in version 2.0.0.

Use server-side encryption for logs stored in S3

Type: string
Default: False
Environment Variable: AIRFLOW__LOGGING__ENCRYPT_S3_LOGS

extra_logger_names¶

New in version 2.0.0.

A comma-separated list of third-party logger names that will be configured to print messages to consoles.

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__EXTRA_LOGGER_NAMES
Example: connexion,sqlalchemy

fab_logging_level¶

New in version 2.0.0.

Logging level for Flask-appbuilder UI.

Supported values: CRITICAL, ERROR, WARNING, INFO, DEBUG.

Type: string
Default: WARNING
Environment Variable: AIRFLOW__LOGGING__FAB_LOGGING_LEVEL

file_task_handler_new_file_permissions¶

New in version 2.6.0.

Permissions in the form or of octal string as understood by chmod. The permissions are important when you use impersonation, when logs are written by a different user than airflow. The most secure way of configuring it in this case is to add both users to the same group and make it the default group of both users. Group-writeable logs are default in airflow, but you might decide that you are OK with having the logs other-writeable, in which case you should set it to 0o666. You might decide to add more security if you do not use impersonation and change it to 0o644 to make it only owner-writeable. You can also make it just readable only for owner by changing it to 0o600 if all the access (read/write) for your logs happens from the same user.

Type: string
Default: 0o664
Environment Variable: AIRFLOW__LOGGING__FILE_TASK_HANDLER_NEW_FILE_PERMISSIONS
Example: 0o664

file_task_handler_new_folder_permissions¶

New in version 2.6.0.

Permissions in the form or of octal string as understood by chmod. The permissions are important when you use impersonation, when logs are written by a different user than airflow. The most secure way of configuring it in this case is to add both users to the same group and make it the default group of both users. Group-writeable logs are default in airflow, but you might decide that you are OK with having the logs other-writeable, in which case you should set it to 0o777. You might decide to add more security if you do not use impersonation and change it to 0o755 to make it only owner-writeable. You can also make it just readable only for owner by changing it to 0o700 if all the access (read/write) for your logs happens from the same user.

Type: string
Default: 0o775
Environment Variable: AIRFLOW__LOGGING__FILE_TASK_HANDLER_NEW_FOLDER_PERMISSIONS
Example: 0o775

google_key_path¶

New in version 2.0.0.

Path to Google Credential JSON file. If omitted, authorization based on the Application Default Credentials will be used.

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__GOOGLE_KEY_PATH

interleave_timestamp_parser¶

New in version 2.6.0.

We must parse timestamps to interleave logs between trigger and task. To do so, we need to parse timestamps in log files. In case your log format is non-standard, you may provide import path to callable which takes a string log line and returns the timestamp (datetime.datetime compatible).

Type: string
Default: None
Environment Variable: AIRFLOW__LOGGING__INTERLEAVE_TIMESTAMP_PARSER
Example: path.to.my_func

log_filename_template¶

New in version 2.0.0.

Formatting for how airflow generates file names/paths for each task run.

Type: string
Default: dag_id={ ti.dag_id }/run_id={ ti.run_id }/task_id={ ti.task_id }/{%% if ti.map_index >= 0 %%}map_index={ ti.map_index }/{%% endif %%}attempt={ try_number }.log
Environment Variable: AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE

log_format¶

New in version 2.0.0.

Format of Log line

Type: string
Default: [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
Environment Variable: AIRFLOW__LOGGING__LOG_FORMAT

log_formatter_class¶

New in version 2.3.4.

Determines the formatter class used by Airflow for structuring its log messages The default formatter class is timezone-aware, which means that timestamps attached to log entries will be adjusted to reflect the local timezone of the Airflow instance

Type: string
Default: airflow.utils.log.timezone_aware.TimezoneAware
Environment Variable: AIRFLOW__LOGGING__LOG_FORMATTER_CLASS

log_processor_filename_template¶

New in version 2.0.0.

Formatting for how airflow generates file names for log

Type: string
Default: { filename }.log
Environment Variable: AIRFLOW__LOGGING__LOG_PROCESSOR_FILENAME_TEMPLATE

logging_config_class¶

New in version 2.0.0.

Logging class Specify the class that will specify the logging configuration This class has to be on the python classpath

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
Example: my.path.default_local_settings.LOGGING_CONFIG

logging_level¶

New in version 2.0.0.

Logging level.

Supported values: CRITICAL, ERROR, WARNING, INFO, DEBUG.

Type: string
Default: INFO
Environment Variable: AIRFLOW__LOGGING__LOGGING_LEVEL

remote_base_log_folder¶

New in version 2.0.0.

Storage bucket URL for remote logging S3 buckets should start with s3:// Cloudwatch log groups should start with cloudwatch:// GCS buckets should start with gs:// WASB buckets should start with wasb just to help Airflow select correct handler Stackdriver logs should start with stackdriver://

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER

remote_log_conn_id¶

New in version 2.0.0.

Users must supply an Airflow connection id that provides access to the storage location. Depending on your remote logging service, this may only be used for reading logs, not writing them.

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID

remote_logging¶

New in version 2.0.0.

Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search. Set this to True if you want to enable remote logging.

Type: string
Default: False
Environment Variable: AIRFLOW__LOGGING__REMOTE_LOGGING

remote_task_handler_kwargs¶

New in version 2.6.0.

The remote_task_handler_kwargs param is loaded into a dictionary and passed to the __init__ of remote task handler and it overrides the values provided by Airflow config. For example if you set delete_local_logs=False and you provide {"delete_local_copy": true}, then the local log files will be deleted after they are uploaded to remote location.

Type

string

Default

''

Environment Variables

AIRFLOW__LOGGING__REMOTE_TASK_HANDLER_KWARGS

AIRFLOW__LOGGING__REMOTE_TASK_HANDLER_KWARGS_CMD

AIRFLOW__LOGGING__REMOTE_TASK_HANDLER_KWARGS_SECRET

Example

{"delete_local_copy": true}

secret_mask_adapter¶

New in version 2.6.0.

An import path to a function to add adaptations of each secret added with airflow.utils.log.secrets_masker.mask_secret to be masked in log messages. The given function is expected to require a single parameter: the secret to be adapted. It may return a single adaptation of the secret or an iterable of adaptations to each be masked as secrets. The original secret will be masked as well as any adaptations returned.

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__SECRET_MASK_ADAPTER
Example: urllib.parse.quote

simple_log_format¶

New in version 2.0.0.

Defines the format of log messages for simple logging configuration

Type: string
Default: %%(asctime)s %%(levelname)s - %%(message)s
Environment Variable: AIRFLOW__LOGGING__SIMPLE_LOG_FORMAT

task_log_prefix_template¶

New in version 2.0.0.

Specify prefix pattern like mentioned below with stream handler TaskHandlerWithCustomFormatter

Type: string
Default: ''
Environment Variable: AIRFLOW__LOGGING__TASK_LOG_PREFIX_TEMPLATE
Example: {ti.dag_id}-{ti.task_id}-{execution_date}-{ti.try_number}

task_log_reader¶

New in version 2.0.0.

Name of handler to read task instance logs. Defaults to use task handler.

Type: string
Default: task
Environment Variable: AIRFLOW__LOGGING__TASK_LOG_READER

trigger_log_server_port¶

New in version 2.6.0.

Port to serve logs from for triggerer. See [logging] worker_log_server_port description for more info.

Type: string
Default: 8794
Environment Variable: AIRFLOW__LOGGING__TRIGGER_LOG_SERVER_PORT

worker_log_server_port¶

New in version 2.2.0.

When you start an Airflow worker, Airflow starts a tiny web server subprocess to serve the workers local log files to the airflow main web server, who then builds pages and sends them to users. This defines the port on which the logs are served. It needs to be unused, and open visible from the main web server to connect into the workers.

Type: string
Default: 8793
Environment Variable: AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT

[metrics]¶

StatsD integration settings.

metrics_allow_list¶

New in version 2.6.0.

Configure an allow list (comma separated string) to send only certain metrics. If [metrics] metrics_use_pattern_match is false, match only the exact metric name prefix. If [metrics] metrics_use_pattern_match is true, provide regex patterns to match.

Type: string
Default: ''
Environment Variable: AIRFLOW__METRICS__METRICS_ALLOW_LIST
Example: "scheduler,executor,dagrun" or "^scheduler,^executor,heartbeat|timeout"

metrics_block_list¶

New in version 2.6.0.

Configure a block list (comma separated string) to block certain metrics from being emitted. If [metrics] metrics_allow_list and [metrics] metrics_block_list are both configured, [metrics] metrics_block_list is ignored.

If [metrics] metrics_use_pattern_match is false, match only the exact metric name prefix.

If [metrics] metrics_use_pattern_match is true, provide regex patterns to match.

Type: string
Default: ''
Environment Variable: AIRFLOW__METRICS__METRICS_BLOCK_LIST
Example: "scheduler,executor,dagrun" or "^scheduler,^executor,heartbeat|timeout"

metrics_use_pattern_match¶

New in version 2.9.0.

If true, [metrics] metrics_allow_list and [metrics] metrics_block_list will use regex pattern matching anywhere within the metric name instead of only prefix matching at the start of the name.

Type: boolean
Default: False
Environment Variable: AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH

otel_debugging_on¶

New in version 2.7.0.

If True, all metrics are also emitted to the console. Defaults to False.

Type: string
Default: False
Environment Variable: AIRFLOW__METRICS__OTEL_DEBUGGING_ON

otel_host¶

New in version 2.6.0.

Specifies the hostname or IP address of the OpenTelemetry Collector to which Airflow sends metrics and traces.

Type: string
Default: localhost
Environment Variable: AIRFLOW__METRICS__OTEL_HOST

otel_interval_milliseconds¶

New in version 2.6.0.

Defines the interval, in milliseconds, at which Airflow sends batches of metrics and traces to the configured OpenTelemetry Collector.

Type: integer
Default: 60000
Environment Variable: AIRFLOW__METRICS__OTEL_INTERVAL_MILLISECONDS

otel_on¶

New in version 2.6.0.

Enables sending metrics to OpenTelemetry.

Type: string
Default: False
Environment Variable: AIRFLOW__METRICS__OTEL_ON

otel_port¶

New in version 2.6.0.

Specifies the port of the OpenTelemetry Collector that is listening to.

Type: string
Default: 8889
Environment Variable: AIRFLOW__METRICS__OTEL_PORT

otel_prefix¶

New in version 2.6.0.

The prefix for the Airflow metrics.

Type: string
Default: airflow
Environment Variable: AIRFLOW__METRICS__OTEL_PREFIX

otel_ssl_active¶

New in version 2.7.0.

If True, SSL will be enabled. Defaults to False. To establish an HTTPS connection to the OpenTelemetry collector, you need to configure the SSL certificate and key within the OpenTelemetry collector’s config.yml file.

Type: string
Default: False
Environment Variable: AIRFLOW__METRICS__OTEL_SSL_ACTIVE

stat_name_handler¶

New in version 2.0.0.

A function that validate the StatsD stat name, apply changes to the stat name if necessary and return the transformed stat name.

The function should have the following signature

def func_name(stat_name: str) -> str: ...

Type: string
Default: ''
Environment Variable: AIRFLOW__METRICS__STAT_NAME_HANDLER

statsd_custom_client_path¶

New in version 2.0.0.

If you want to utilise your own custom StatsD client set the relevant module path below. Note: The module path must exist on your PYTHONPATH <https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH> for Airflow to pick it up

Type: string
Default: None
Environment Variable: AIRFLOW__METRICS__STATSD_CUSTOM_CLIENT_PATH

statsd_datadog_enabled¶

New in version 2.0.0.

To enable datadog integration to send airflow metrics.

Type: string
Default: False
Environment Variable: AIRFLOW__METRICS__STATSD_DATADOG_ENABLED

statsd_datadog_metrics_tags¶

New in version 2.6.0.

Set to False to disable metadata tags for some of the emitted metrics

Type: boolean
Default: True
Environment Variable: AIRFLOW__METRICS__STATSD_DATADOG_METRICS_TAGS

statsd_datadog_tags¶

New in version 2.0.0.

List of datadog tags attached to all metrics(e.g: key1:value1,key2:value2)

Type: string
Default: ''
Environment Variable: AIRFLOW__METRICS__STATSD_DATADOG_TAGS

statsd_disabled_tags¶

New in version 2.6.0.

If you want to avoid sending all the available metrics tags to StatsD, you can configure a block list of prefixes (comma separated) to filter out metric tags that start with the elements of the list (e.g: job_id,run_id)

Type: string
Default: job_id,run_id
Environment Variable: AIRFLOW__METRICS__STATSD_DISABLED_TAGS
Example: job_id,run_id,dag_id,task_id

statsd_host¶

New in version 2.0.0.

Specifies the host address where the StatsD daemon (or server) is running

Type: string
Default: localhost
Environment Variable: AIRFLOW__METRICS__STATSD_HOST

statsd_influxdb_enabled¶

New in version 2.6.0.

To enable sending Airflow metrics with StatsD-Influxdb tagging convention.

Type: boolean
Default: False
Environment Variable: AIRFLOW__METRICS__STATSD_INFLUXDB_ENABLED

statsd_on¶

New in version 2.0.0.

Enables sending metrics to StatsD.

Type: string
Default: False
Environment Variable: AIRFLOW__METRICS__STATSD_ON

statsd_port¶

New in version 2.0.0.

Specifies the port on which the StatsD daemon (or server) is listening to

Type: string
Default: 8125
Environment Variable: AIRFLOW__METRICS__STATSD_PORT

statsd_prefix¶

New in version 2.0.0.

Defines the namespace for all metrics sent from Airflow to StatsD

Type: string
Default: airflow
Environment Variable: AIRFLOW__METRICS__STATSD_PREFIX

statsd_allow_list (Deprecated)¶

Deprecated since version 2.6.0: The option has been moved to metrics.metrics_allow_list

statsd_block_list (Deprecated)¶

Deprecated since version 2.6.0: The option has been moved to metrics.metrics_block_list

[operators]¶

allow_illegal_arguments¶

New in version 2.0.0.

Is allowed to pass additional/unused arguments (args, kwargs) to the BaseOperator operator. If set to False, an exception will be thrown, otherwise only the console message will be displayed.

Type: string
Default: False
Environment Variable: AIRFLOW__OPERATORS__ALLOW_ILLEGAL_ARGUMENTS

default_cpus¶

Indicates the default number of CPU units allocated to each operator when no specific CPU request is specified in the operator’s configuration

Type: string
Default: 1
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_CPUS

default_deferrable¶

New in version 2.7.0.

The default value of attribute “deferrable” in operators and sensors.

Type: boolean
Default: false
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE

default_disk¶

Indicates the default number of disk storage allocated to each operator when no specific disk request is specified in the operator’s configuration

Type: string
Default: 512
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_DISK

default_gpus¶

Indicates the default number of GPUs allocated to each operator when no specific GPUs request is specified in the operator’s configuration

Type: string
Default: 0
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_GPUS

default_owner¶

The default owner assigned to each new operator, unless provided explicitly or passed via default_args

Type: string
Default: airflow
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_OWNER

default_queue¶

New in version 2.1.0.

Default queue that tasks get assigned to and that worker listen on.

Type: string
Default: default
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_QUEUE

default_ram¶

Indicates the default number of RAM allocated to each operator when no specific RAM request is specified in the operator’s configuration

Type: string
Default: 512
Environment Variable: AIRFLOW__OPERATORS__DEFAULT_RAM

[scheduler]¶

allow_trigger_in_future¶

New in version 1.10.8.

Allow externally triggered DagRuns for Execution Dates in the future Only has effect if schedule_interval is set to None in DAG

Type: boolean
Default: False
Environment Variable: AIRFLOW__SCHEDULER__ALLOW_TRIGGER_IN_FUTURE

allowed_run_id_pattern¶

New in version 2.6.3.

The run_id pattern used to verify the validity of user input to the run_id parameter when triggering a DAG. This pattern cannot change the pattern used by scheduler to generate run_id for scheduled DAG runs or DAG runs triggered without changing the run_id parameter.

Type: string
Default: ^[A-Za-z0-9_.~:+-]+$
Environment Variable: AIRFLOW__SCHEDULER__ALLOWED_RUN_ID_PATTERN

catchup_by_default¶

Turn off scheduler catchup by setting this to False. Default behavior is unchanged and Command Line Backfills still work, but the scheduler will not do scheduler catchup if this is False, however it can be set on a per DAG basis in the DAG definition (catchup)

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT

child_process_log_directory¶

Determines the directory where logs for the child processes of the scheduler will be stored

Type: string
Default: {AIRFLOW_HOME}/logs/scheduler
Environment Variable: AIRFLOW__SCHEDULER__CHILD_PROCESS_LOG_DIRECTORY

create_cron_data_intervals¶

New in version 2.9.0.

Whether to create DAG runs that span an interval or one single point in time for cron schedules, when a cron string is provided to schedule argument of a DAG.

True: CronDataIntervalTimetable is used, which is suitable for DAGs with well-defined data interval. You get contiguous intervals from the end of the previous interval up to the scheduled datetime.
False: CronTriggerTimetable is used, which is closer to the behavior of cron itself.

Notably, for CronTriggerTimetable, the logical date is the same as the time the DAG Run will try to schedule, while for CronDataIntervalTimetable, the logical date is the beginning of the data interval, but the DAG Run will try to schedule at the end of the data interval.

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS

dag_dir_list_interval¶

How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.

Type: integer
Default: 300
Environment Variable: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL

dag_stale_not_seen_duration¶

New in version 2.4.0.

Only applicable if [scheduler] standalone_dag_processor is true. Time in seconds after which dags, which were not updated by Dag Processor are deactivated.

Type: integer
Default: 600
Environment Variable: AIRFLOW__SCHEDULER__DAG_STALE_NOT_SEEN_DURATION

enable_health_check¶

New in version 2.4.0.

When you start a scheduler, airflow starts a tiny web server subprocess to serve a health check if this is set to True

Type: boolean
Default: False
Environment Variable: AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK

file_parsing_sort_mode¶

New in version 2.1.0.

One of modified_time, random_seeded_by_host and alphabetical. The scheduler will list and sort the dag files to decide the parsing order.

modified_time: Sort by modified time of the files. This is useful on large scale to parse the recently modified DAGs first.
random_seeded_by_host: Sort randomly across multiple Schedulers but with same order on the same host. This is useful when running with Scheduler in HA mode where each scheduler can parse different DAG files.
alphabetical: Sort by filename

Type: string
Default: modified_time
Environment Variable: AIRFLOW__SCHEDULER__FILE_PARSING_SORT_MODE

ignore_first_depends_on_past_by_default¶

New in version 2.3.0.

Setting this to True will make first task instance of a task ignore depends_on_past setting. A task instance will be considered as the first task instance of a task when there is no task instance in the DB with an execution_date earlier than it., i.e. no manual marking success will be needed for a newly added task to be scheduled.

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__IGNORE_FIRST_DEPENDS_ON_PAST_BY_DEFAULT

job_heartbeat_sec¶

Task instances listen for external kill signal (when you clear tasks from the CLI or the UI), this defines the frequency at which they should listen (in seconds).

Type: float
Default: 5
Environment Variable: AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC

local_task_job_heartbeat_sec¶

New in version 2.7.0.

The frequency (in seconds) at which the LocalTaskJob should send heartbeat signals to the scheduler to notify it’s still alive. If this value is set to 0, the heartbeat interval will default to the value of [scheduler] scheduler_zombie_task_threshold.

Type: integer
Default: 0
Environment Variable: AIRFLOW__SCHEDULER__LOCAL_TASK_JOB_HEARTBEAT_SEC

max_callbacks_per_loop¶

New in version 2.3.0.

Only applicable if [scheduler] standalone_dag_processor is true and callbacks are stored in database. Contains maximum number of callbacks that are fetched during a single loop.

Type: integer
Default: 20
Environment Variable: AIRFLOW__SCHEDULER__MAX_CALLBACKS_PER_LOOP

max_dagruns_per_loop_to_schedule¶

New in version 2.0.0.

How many DagRuns should a scheduler examine (and lock) when scheduling and queuing tasks.

See also

Scheduler Configuration options

Type: integer
Default: 20
Environment Variable: AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE

max_dagruns_to_create_per_loop¶

New in version 2.0.0.

Max number of DAGs to create DagRuns for per scheduler loop.

See also

Scheduler Configuration options

Type: integer
Default: 10
Environment Variable: AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP

max_tis_per_query¶

This changes the batch size of queries in the scheduling main loop. This should not be greater than [core] parallelism. If this is too high, SQL query performance may be impacted by complexity of query predicate, and/or excessive locking. Additionally, you may hit the maximum allowable query length for your db. Set this to 0 to use the value of [core] parallelism

Type: integer
Default: 16
Environment Variable: AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY

min_file_process_interval¶

Number of seconds after which a DAG file is parsed. The DAG file is parsed every [scheduler] min_file_process_interval number of seconds. Updates to DAGs are reflected after this interval. Keeping this number low will increase CPU usage.

Type: integer
Default: 30
Environment Variable: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL

num_runs¶

New in version 1.10.6.

The number of times to try to schedule each DAG file -1 indicates unlimited number

Type: integer
Default: -1
Environment Variable: AIRFLOW__SCHEDULER__NUM_RUNS

orphaned_tasks_check_interval¶

New in version 2.0.0.

How often (in seconds) should the scheduler check for orphaned tasks and SchedulerJobs

Type: float
Default: 300.0
Environment Variable: AIRFLOW__SCHEDULER__ORPHANED_TASKS_CHECK_INTERVAL

parsing_cleanup_interval¶

New in version 2.5.0.

How often (in seconds) to check for stale DAGs (DAGs which are no longer present in the expected files) which should be deactivated, as well as datasets that are no longer referenced and should be marked as orphaned.

Type: integer
Default: 60
Environment Variable: AIRFLOW__SCHEDULER__PARSING_CLEANUP_INTERVAL

parsing_pre_import_modules¶

New in version 2.6.0.

The scheduler reads dag files to extract the airflow modules that are going to be used, and imports them ahead of time to avoid having to re-do it for each parsing process. This flag can be set to False to disable this behavior in case an airflow module needs to be freshly imported each time (at the cost of increased DAG parsing time).

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__PARSING_PRE_IMPORT_MODULES

parsing_processes¶

New in version 1.10.14.

The scheduler can run multiple processes in parallel to parse dags. This defines how many processes will run.

Type: integer
Default: 2
Environment Variable: AIRFLOW__SCHEDULER__PARSING_PROCESSES

pool_metrics_interval¶

New in version 2.0.0.

How often (in seconds) should pool usage stats be sent to StatsD (if statsd_on is enabled)

Type: float
Default: 5.0
Environment Variable: AIRFLOW__SCHEDULER__POOL_METRICS_INTERVAL

print_stats_interval¶

How often should stats be printed to the logs. Setting to 0 will disable printing stats

Type: integer
Default: 30
Environment Variable: AIRFLOW__SCHEDULER__PRINT_STATS_INTERVAL

schedule_after_task_execution¶

New in version 2.0.0.

Should the Task supervisor process perform a “mini scheduler” to attempt to schedule more tasks of the same DAG. Leaving this on will mean tasks in the same DAG execute quicker, but might starve out other dags in some circumstances

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION

scheduler_health_check_server_host¶

New in version 2.8.0.

When you start a scheduler, airflow starts a tiny web server subprocess to serve a health check on this host

Type: string
Default: 0.0.0.0
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_SERVER_HOST

scheduler_health_check_server_port¶

New in version 2.4.0.

When you start a scheduler, airflow starts a tiny web server subprocess to serve a health check on this port

Type: integer
Default: 8974
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_SERVER_PORT

scheduler_health_check_threshold¶

New in version 1.10.2.

If the last scheduler heartbeat happened more than [scheduler] scheduler_health_check_threshold ago (in seconds), scheduler is considered unhealthy. This is used by the health check in the /health endpoint and in airflow jobs check CLI for SchedulerJob.

Type: integer
Default: 30
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_THRESHOLD

scheduler_heartbeat_sec¶

The scheduler constantly tries to trigger new tasks (look at the scheduler section in the docs for more information). This defines how often the scheduler should run (in seconds).

Type: integer
Default: 5
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC

scheduler_idle_sleep_time¶

New in version 2.2.0.

Controls how long the scheduler will sleep between loops, but if there was nothing to do in the loop. i.e. if it scheduled something then it will start the next loop iteration straight away.

Type: float
Default: 1
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULER_IDLE_SLEEP_TIME

scheduler_zombie_task_threshold¶

Local task jobs periodically heartbeat to the DB. If the job has not heartbeat in this many seconds, the scheduler will mark the associated task instance as failed and will re-schedule the task.

Type: integer
Default: 300
Environment Variable: AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD

stale_dag_threshold¶

New in version 2.6.0.

How long (in seconds) to wait after we have re-parsed a DAG file before deactivating stale DAGs (DAGs which are no longer present in the expected files). The reason why we need this threshold is to account for the time between when the file is parsed and when the DAG is loaded. The absolute maximum that this could take is [core] dag_file_processor_timeout, but when you have a long timeout configured, it results in a significant delay in the deactivation of stale dags.

Type: integer
Default: 50
Environment Variable: AIRFLOW__SCHEDULER__STALE_DAG_THRESHOLD

standalone_dag_processor¶

New in version 2.3.0.

Whether the dag processor is running as a standalone process or it is a subprocess of a scheduler job.

Type: boolean
Default: False
Environment Variable: AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR

task_queued_timeout¶

New in version 2.6.0.

Amount of time a task can be in the queued state before being retried or set to failed.

Type: float
Default: 600.0
Environment Variable: AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT

task_queued_timeout_check_interval¶

New in version 2.6.0.

How often to check for tasks that have been in the queued state for longer than [scheduler] task_queued_timeout.

Type: float
Default: 120.0
Environment Variable: AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT_CHECK_INTERVAL

trigger_timeout_check_interval¶

New in version 2.2.0.

How often to check for expired trigger requests that have not run yet.

Type: float
Default: 15
Environment Variable: AIRFLOW__SCHEDULER__TRIGGER_TIMEOUT_CHECK_INTERVAL

use_job_schedule¶

New in version 1.10.2.

Turn off scheduler use of cron intervals by setting this to False. DAGs submitted manually in the web UI or with trigger_dag will still run.

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE

use_row_level_locking¶

New in version 2.0.0.

Should the scheduler issue SELECT ... FOR UPDATE in relevant queries. If this is set to False then you should not run more than a single scheduler at once

Type: boolean
Default: True
Environment Variable: AIRFLOW__SCHEDULER__USE_ROW_LEVEL_LOCKING

zombie_detection_interval¶

New in version 2.3.0.

How often (in seconds) should the scheduler check for zombie tasks.

Type: float
Default: 10.0
Environment Variable: AIRFLOW__SCHEDULER__ZOMBIE_DETECTION_INTERVAL

deactivate_stale_dags_interval (Deprecated)¶

Deprecated since version 2.5.0: The option has been moved to scheduler.parsing_cleanup_interval

max_threads (Deprecated)¶

Deprecated since version 1.10.14: The option has been moved to scheduler.parsing_processes

processor_poll_interval (Deprecated)¶

Deprecated since version 2.2.0: The option has been moved to scheduler.scheduler_idle_sleep_time

stat_name_handler (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.stat_name_handler

statsd_allow_list (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_allow_list

statsd_custom_client_path (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_custom_client_path

statsd_datadog_enabled (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_datadog_enabled

statsd_datadog_metrics_tags (Deprecated)¶

Deprecated since version 2.6.0: The option has been moved to metrics.statsd_datadog_metrics_tags

statsd_datadog_tags (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_datadog_tags

statsd_host (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_host

statsd_on (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_on

statsd_port (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_port

statsd_prefix (Deprecated)¶

Deprecated since version 2.0.0: The option has been moved to metrics.statsd_prefix

[secrets]¶

backend¶

New in version 1.10.10.

Full class name of secrets backend to enable (will precede env vars and metastore in search path)

Type: string
Default: ''
Environment Variable: AIRFLOW__SECRETS__BACKEND
Example: airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend

backend_kwargs¶

New in version 1.10.10.

The backend_kwargs param is loaded into a dictionary and passed to __init__ of secrets backend class. See documentation for the secrets backend you are using. JSON is expected.

Example for AWS Systems Manager ParameterStore: {"connections_prefix": "/airflow/connections", "profile_name": "default"}

Type

string

Default

''

Environment Variables

AIRFLOW__SECRETS__BACKEND_KWARGS

AIRFLOW__SECRETS__BACKEND_KWARGS_CMD

AIRFLOW__SECRETS__BACKEND_KWARGS_SECRET

cache_ttl_seconds¶

New in version 2.7.0.

Note

This is an experimental feature.

When the cache is enabled, this is the duration for which we consider an entry in the cache to be valid. Entries are refreshed if they are older than this many seconds. It means that when the cache is enabled, this is the maximum amount of time you need to wait to see a Variable change take effect.

Type: integer
Default: 900
Environment Variable: AIRFLOW__SECRETS__CACHE_TTL_SECONDS

use_cache¶

New in version 2.7.0.

Note

This is an experimental feature.

Enables local caching of Variables, when parsing DAGs only. Using this option can make dag parsing faster if Variables are used in top level code, at the expense of longer propagation time for changes. Please note that this cache concerns only the DAG parsing step. There is no caching in place when DAG tasks are run.

Type: boolean
Default: False
Environment Variable: AIRFLOW__SECRETS__USE_CACHE

[sensors]¶

default_timeout¶

New in version 2.3.0.

Sensor default timeout, 7 days by default (7 * 24 * 60 * 60).

Type: float
Default: 604800
Environment Variable: AIRFLOW__SENSORS__DEFAULT_TIMEOUT

[sentry]¶

Sentry integration. Here you can supply additional configuration options based on the Python platform. See Python / Configuration / Basic Options for more details. Unsupported options: integrations, in_app_include, in_app_exclude, ignore_errors, before_breadcrumb, transport.

before_send¶

New in version 2.2.0.

Dotted path to a before_send function that the sentry SDK should be configured to use.

Type: string
Default: None
Environment Variable: AIRFLOW__SENTRY__BEFORE_SEND

sentry_dsn¶

New in version 1.10.6.

Type

string

Default

''

Environment Variables

AIRFLOW__SENTRY__SENTRY_DSN

AIRFLOW__SENTRY__SENTRY_DSN_CMD

AIRFLOW__SENTRY__SENTRY_DSN_SECRET

sentry_on¶

New in version 2.0.0.

Enable error reporting to Sentry

Type: string
Default: false
Environment Variable: AIRFLOW__SENTRY__SENTRY_ON

[smtp]¶

If you want airflow to send emails on retries, failure, and you want to use the airflow.utils.email.send_email_smtp function, you have to configure an smtp server here

smtp_host¶

Specifies the host server address used by Airflow when sending out email notifications via SMTP.

Type: string
Default: localhost
Environment Variable: AIRFLOW__SMTP__SMTP_HOST

smtp_mail_from¶

Specifies the default from email address used when Airflow sends email notifications.

Type: string
Default: airflow@example.com
Environment Variable: AIRFLOW__SMTP__SMTP_MAIL_FROM

smtp_password¶

Password to authenticate when connecting to smtp server.

Type

string

Default

None

Environment Variables

AIRFLOW__SMTP__SMTP_PASSWORD

AIRFLOW__SMTP__SMTP_PASSWORD_CMD

AIRFLOW__SMTP__SMTP_PASSWORD_SECRET

Example

airflow

smtp_port¶

Defines the port number on which Airflow connects to the SMTP server to send email notifications.

Type: string
Default: 25
Environment Variable: AIRFLOW__SMTP__SMTP_PORT

smtp_retry_limit¶

New in version 2.0.0.

Defines the maximum number of times Airflow will attempt to connect to the SMTP server.

Type: integer
Default: 5
Environment Variable: AIRFLOW__SMTP__SMTP_RETRY_LIMIT

smtp_ssl¶

Determines whether to use an SSL connection when talking to the SMTP server.

Type: string
Default: False
Environment Variable: AIRFLOW__SMTP__SMTP_SSL

smtp_starttls¶

Determines whether to use the STARTTLS command when connecting to the SMTP server.

Type: string
Default: True
Environment Variable: AIRFLOW__SMTP__SMTP_STARTTLS

smtp_timeout¶

New in version 2.0.0.

Determines the maximum time (in seconds) the Apache Airflow system will wait for a connection to the SMTP server to be established.

Type: integer
Default: 30
Environment Variable: AIRFLOW__SMTP__SMTP_TIMEOUT

smtp_user¶

Username to authenticate when connecting to smtp server.

Type: string
Default: None
Environment Variable: AIRFLOW__SMTP__SMTP_USER
Example: airflow

[triggerer]¶

default_capacity¶

New in version 2.2.0.

How many triggers a single Triggerer will run at once, by default.

Type: string
Default: 1000
Environment Variable: AIRFLOW__TRIGGERER__DEFAULT_CAPACITY

job_heartbeat_sec¶

New in version 2.6.3.

How often to heartbeat the Triggerer job to ensure it hasn’t been killed.

Type: float
Default: 5
Environment Variable: AIRFLOW__TRIGGERER__JOB_HEARTBEAT_SEC

triggerer_health_check_threshold¶

New in version 2.7.0.

If the last triggerer heartbeat happened more than [triggerer] triggerer_health_check_threshold ago (in seconds), triggerer is considered unhealthy. This is used by the health check in the /health endpoint and in airflow jobs check CLI for TriggererJob.

Type: float
Default: 30
Environment Variable: AIRFLOW__TRIGGERER__TRIGGERER_HEALTH_CHECK_THRESHOLD

[webserver]¶

access_denied_message¶

New in version 2.7.0.

The message displayed when a user attempts to execute actions beyond their authorised privileges.

Type: string
Default: Access is Denied
Environment Variable: AIRFLOW__WEBSERVER__ACCESS_DENIED_MESSAGE

access_logfile¶

Log files for the gunicorn webserver. ‘-‘ means log to stderr.

Type: string
Default: -
Environment Variable: AIRFLOW__WEBSERVER__ACCESS_LOGFILE

access_logformat¶

New in version 2.0.0.

Access log format for gunicorn webserver. default format is %%(h)s %%(l)s %%(u)s %%(t)s "%%(r)s" %%(s)s %%(b)s "%%(f)s" "%%(a)s" See Gunicorn Settings: ‘access_log_format’ Reference for more details

Type: string
Default: ''
Environment Variable: AIRFLOW__WEBSERVER__ACCESS_LOGFORMAT

allow_raw_html_descriptions¶

New in version 2.8.0.

A DAG author is able to provide any raw HTML into doc_md or params description in description_md for text formatting. This is including potentially unsafe javascript. Displaying the DAG or trigger form in web UI provides the DAG author the potential to inject malicious code into clients browsers. To ensure the web UI is safe by default, raw HTML is disabled by default. If you trust your DAG authors, you can enable HTML support in markdown by setting this option to True.

This parameter also enables the deprecated fields description_html and custom_html_form in DAG params until the feature is removed in a future version.

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__ALLOW_RAW_HTML_DESCRIPTIONS
Example: False

allowed_payload_size¶

New in version 2.8.1.

The maximum size of the request payload (in MB) that can be sent.

Type: float
Default: 1.0
Environment Variable: AIRFLOW__WEBSERVER__ALLOWED_PAYLOAD_SIZE

analytics_id¶

New in version 1.10.5.

Unique ID of your account in the analytics tool

Type: string
Default: None
Environment Variable: AIRFLOW__WEBSERVER__ANALYTICS_ID

analytics_tool¶

Send anonymous user activity to your analytics tool choose from google_analytics, segment, metarouter, or matomo

Type: string
Default: None
Environment Variable: AIRFLOW__WEBSERVER__ANALYTICS_TOOL

analytics_url¶

New in version 2.9.0.

Your instances url, only applicable to Matomo.

Type: string
Default: None
Environment Variable: AIRFLOW__WEBSERVER__ANALYTICS_URL
Example: https://your.matomo.instance.com/

audit_view_excluded_events¶

New in version 2.3.0.

Comma separated string of view events to exclude from dag audit view. All other events will be added minus the ones passed here. The audit logs in the db will not be affected by this parameter.

Type: string
Default: None
Environment Variable: AIRFLOW__WEBSERVER__AUDIT_VIEW_EXCLUDED_EVENTS
Example: cli_task_run,running,success

audit_view_included_events¶

New in version 2.3.0.

Comma separated string of view events to include in dag audit view. If passed, only these events will populate the dag audit view. The audit logs in the db will not be affected by this parameter.

Type: string
Default: None
Environment Variable: AIRFLOW__WEBSERVER__AUDIT_VIEW_INCLUDED_EVENTS
Example: dagrun_cleared,failed

auth_rate_limit¶

New in version 2.6.0.

Rate limit for authentication endpoints.

Deprecated since version 2.9.0: This config has been moved to fab provider. Please use the config from fab provider.

Type: string
Default: 5 per 40 second
Environment Variable: AIRFLOW__WEBSERVER__AUTH_RATE_LIMIT

auth_rate_limited¶

New in version 2.6.0.

Boolean for enabling rate limiting on authentication endpoints.

Deprecated since version 2.9.0: This config has been moved to fab provider. Please use the config from fab provider.

Type: boolean
Default: True
Environment Variable: AIRFLOW__WEBSERVER__AUTH_RATE_LIMITED

auto_refresh_interval¶

New in version 2.2.0.

How frequently, in seconds, the DAG data will auto-refresh in graph or grid view when auto-refresh is turned on

Type: integer
Default: 3
Environment Variable: AIRFLOW__WEBSERVER__AUTO_REFRESH_INTERVAL

base_url¶

The base url of your website: Airflow cannot guess what domain or CNAME you are using. This is used to create links in the Log Url column in the Browse - Task Instances menu, as well as in any automated emails sent by Airflow that contain links to your webserver.

Type: string
Default: http://localhost:8080
Environment Variable: AIRFLOW__WEBSERVER__BASE_URL

caching_hash_method¶

New in version 2.6.0.

The caching algorithm used by the webserver. Must be a valid hashlib function name.

Type: string
Default: md5
Environment Variable: AIRFLOW__WEBSERVER__CACHING_HASH_METHOD
Example: sha256

config_file¶

New in version 2.7.0.

Path of webserver config file used for configuring the webserver parameters

Type: string
Default: {AIRFLOW_HOME}/webserver_config.py
Environment Variable: AIRFLOW__WEBSERVER__CONFIG_FILE

dag_default_view¶

Default DAG view. Valid values are: grid, graph, duration, gantt, landing_times

Type: string
Default: grid
Environment Variable: AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW

dag_orientation¶

Default DAG orientation. Valid values are: LR (Left->Right), TB (Top->Bottom), RL (Right->Left), BT (Bottom->Top)

Type: string
Default: LR
Environment Variable: AIRFLOW__WEBSERVER__DAG_ORIENTATION

default_dag_run_display_number¶

Default dagrun to show in UI

Type: string
Default: 25
Environment Variable: AIRFLOW__WEBSERVER__DEFAULT_DAG_RUN_DISPLAY_NUMBER

default_ui_timezone¶

New in version 1.10.10.

Default timezone to display all dates in the UI, can be UTC, system, or any IANA timezone string (e.g. Europe/Amsterdam). If left empty the default value of core/default_timezone will be used

Type: string
Default: UTC
Environment Variable: AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE
Example: America/New_York

default_wrap¶

New in version 1.10.4.

Default setting for wrap toggle on DAG code and TI log views.

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__DEFAULT_WRAP

enable_proxy_fix¶

New in version 1.10.1.

Enable werkzeug ProxyFix middleware for reverse proxy

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX

enable_swagger_ui¶

New in version 2.6.0.

Boolean for running SwaggerUI in the webserver.

Type: boolean
Default: True
Environment Variable: AIRFLOW__WEBSERVER__ENABLE_SWAGGER_UI

error_logfile¶

Log files for the gunicorn webserver. ‘-‘ means log to stderr.

Type: string
Default: -
Environment Variable: AIRFLOW__WEBSERVER__ERROR_LOGFILE

expose_config¶

Expose the configuration file in the web server. Set to non-sensitive-only to show all values except those that have security implications. True shows all values. False hides the configuration completely.

Type: string
Default: False
Environment Variable: AIRFLOW__WEBSERVER__EXPOSE_CONFIG

expose_hostname¶

New in version 1.10.8.

Expose hostname in the web server

Type: string
Default: False
Environment Variable: AIRFLOW__WEBSERVER__EXPOSE_HOSTNAME

expose_stacktrace¶

New in version 1.10.8.

Expose stacktrace in the web server

Type: string
Default: False
Environment Variable: AIRFLOW__WEBSERVER__EXPOSE_STACKTRACE

grid_view_sorting_order¶

New in version 2.7.0.

Sorting order in grid view. Valid values are: topological, hierarchical_alphabetical

Type: string
Default: topological
Environment Variable: AIRFLOW__WEBSERVER__GRID_VIEW_SORTING_ORDER

hide_paused_dags_by_default¶

By default, the webserver shows paused DAGs. Flip this to hide paused DAGs by default

Type: string
Default: False
Environment Variable: AIRFLOW__WEBSERVER__HIDE_PAUSED_DAGS_BY_DEFAULT

instance_name¶

New in version 2.1.0.

Sets a custom page title for the DAGs overview page and site title for all pages

Type: string
Default: None
Environment Variable: AIRFLOW__WEBSERVER__INSTANCE_NAME

instance_name_has_markup¶

New in version 2.3.0.

Whether the custom page title for the DAGs overview page contains any Markup language

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__INSTANCE_NAME_HAS_MARKUP

log_animation_speed¶

New in version 1.10.8.

Animation speed for auto tailing log display.

Type: integer
Default: 1000
Environment Variable: AIRFLOW__WEBSERVER__LOG_ANIMATION_SPEED

log_auto_tailing_offset¶

New in version 1.10.8.

Distance away from page bottom to enable auto tailing.

Type: integer
Default: 30
Environment Variable: AIRFLOW__WEBSERVER__LOG_AUTO_TAILING_OFFSET

log_fetch_delay_sec¶

New in version 1.10.8.

Time interval (in secs) to wait before next log fetching.

Type: integer
Default: 2
Environment Variable: AIRFLOW__WEBSERVER__LOG_FETCH_DELAY_SEC

log_fetch_timeout_sec¶

The amount of time (in secs) webserver will wait for initial handshake while fetching logs from other worker machine

Type: string
Default: 5
Environment Variable: AIRFLOW__WEBSERVER__LOG_FETCH_TIMEOUT_SEC

num_recent_configurations_for_trigger¶

New in version 2.9.0.

Number of recent DAG run configurations in the selector on the trigger web form.

Type: integer
Default: 5
Environment Variable: AIRFLOW__WEBSERVER__NUM_RECENT_CONFIGURATIONS_FOR_TRIGGER
Example: 10

page_size¶

Consistent page size across all listing views in the UI

Type: string
Default: 100
Environment Variable: AIRFLOW__WEBSERVER__PAGE_SIZE

proxy_fix_x_for¶

New in version 1.10.7.

Number of values to trust for X-Forwarded-For. See Werkzeug: X-Forwarded-For Proxy Fix for more details.

Type: integer
Default: 1
Environment Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_FOR

proxy_fix_x_host¶

New in version 1.10.7.

Number of values to trust for X-Forwarded-Host. See Werkzeug: X-Forwarded-For Proxy Fix for more details.

Type: integer
Default: 1
Environment Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_HOST

proxy_fix_x_port¶

New in version 1.10.7.

Number of values to trust for X-Forwarded-Port. See Werkzeug: X-Forwarded-For Proxy Fix for more details.

Type: integer
Default: 1
Environment Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_PORT

proxy_fix_x_prefix¶

New in version 1.10.7.

Number of values to trust for X-Forwarded-Prefix. See Werkzeug: X-Forwarded-For Proxy Fix for more details.

Type: integer
Default: 1
Environment Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_PREFIX

proxy_fix_x_proto¶

New in version 1.10.7.

Number of values to trust for X-Forwarded-Proto. See Werkzeug: X-Forwarded-For Proxy Fix for more details.

Type: integer
Default: 1
Environment Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_PROTO

reload_on_plugin_change¶

New in version 1.10.11.

If set to True, Airflow will track files in plugins_folder directory. When it detects changes, then reload the gunicorn. If set to True, gunicorn starts without preloading, which is slower, uses more memory, and may cause race conditions. Avoid setting this to True in production.

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE

require_confirmation_dag_change¶

New in version 2.9.0.

Require confirmation when changing a DAG in the web UI. This is to prevent accidental changes to a DAG that may be running on sensitive environments like production. When set to True, confirmation dialog will be shown when a user tries to Pause/Unpause, Trigger a DAG

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__REQUIRE_CONFIRMATION_DAG_CHANGE

run_internal_api¶

New in version 2.6.0.

Boolean for running Internal API in the webserver.

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__RUN_INTERNAL_API

secret_key¶

Secret key used to run your flask app. It should be as random as possible. However, when running more than 1 instances of webserver, make sure all of them use the same secret_key otherwise one of them will error with “CSRF session token is missing”. The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token generated using the secret key has a short expiry time though - make sure that time on ALL the machines that you run airflow components on is synchronized (for example using ntpd) otherwise you might get “forbidden” errors when the logs are accessed.

Type

string

Default

{SECRET_KEY}

Environment Variables

AIRFLOW__WEBSERVER__SECRET_KEY

AIRFLOW__WEBSERVER__SECRET_KEY_CMD

AIRFLOW__WEBSERVER__SECRET_KEY_SECRET

session_backend¶

New in version 2.2.4.

The type of backend used to store web session data, can be database or securecookie. For the database backend, sessions are store in the database and they can be managed there (for example when you reset password of the user, all sessions for that user are deleted). For the securecookie backend, sessions are stored in encrypted cookies on the client side. The securecookie mechanism is ‘lighter’ than database backend, but sessions are not deleted when you reset password of the user, which means that other than waiting for expiry time, the only way to invalidate all sessions for a user is to change secret_key and restart webserver (which also invalidates and logs out all other user’s sessions).

When you are using database backend, make sure to keep your database session table small by periodically running airflow db clean --table session command, especially if you have automated API calls that will create a new session for each call rather than reuse the sessions stored in browser cookies.

Type: string
Default: database
Environment Variable: AIRFLOW__WEBSERVER__SESSION_BACKEND
Example: securecookie

session_lifetime_minutes¶

New in version 1.10.13.

The UI cookie lifetime in minutes. User will be logged out from UI after [webserver] session_lifetime_minutes of non-activity

Type: integer
Default: 43200
Environment Variable: AIRFLOW__WEBSERVER__SESSION_LIFETIME_MINUTES

show_recent_stats_for_completed_runs¶

New in version 2.0.0.

‘Recent Tasks’ stats will show for old DagRuns if set

Type: boolean
Default: True
Environment Variable: AIRFLOW__WEBSERVER__SHOW_RECENT_STATS_FOR_COMPLETED_RUNS

show_trigger_form_if_no_params¶

New in version 2.7.0.

Behavior of the trigger DAG run button for DAGs without params. False to skip and trigger without displaying a form to add a dag_run.conf, True to always display the form. The form is displayed always if parameters are defined.

Type: boolean
Default: False
Environment Variable: AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS

update_fab_perms¶

New in version 1.10.7.

Update FAB permissions and sync security manager roles on webserver startup

Deprecated since version 2.9.0: This config has been moved to fab provider. Please use the config from fab provider.

Type: string
Default: True
Environment Variable: AIRFLOW__WEBSERVER__UPDATE_FAB_PERMS

warn_deployment_exposure¶

New in version 2.3.0.

Boolean for displaying warning for publicly viewable deployment

Type: boolean
Default: True
Environment Variable: AIRFLOW__WEBSERVER__WARN_DEPLOYMENT_EXPOSURE

web_server_host¶

The ip specified when starting the web server

Type: string
Default: 0.0.0.0
Environment Variable: AIRFLOW__WEBSERVER__WEB_SERVER_HOST

web_server_master_timeout¶

Number of seconds the webserver waits before killing gunicorn master that doesn’t respond

Type: string
Default: 120
Environment Variable: AIRFLOW__WEBSERVER__WEB_SERVER_MASTER_TIMEOUT

web_server_port¶

The port on which to run the web server

Type: string
Default: 8080
Environment Variable: AIRFLOW__WEBSERVER__WEB_SERVER_PORT

web_server_ssl_cert¶

Paths to the SSL certificate and key for the web server. When both are provided SSL will be enabled. This does not change the web server port.

Type: string
Default: ''
Environment Variable: AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT

web_server_ssl_key¶

Paths to the SSL certificate and key for the web server. When both are provided SSL will be enabled. This does not change the web server port.

Type: string
Default: ''
Environment Variable: AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY

web_server_worker_timeout¶

Number of seconds the gunicorn webserver waits before timing out on a worker

Type: string
Default: 120
Environment Variable: AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT

worker_class¶

The worker class gunicorn should use. Choices include sync (default), eventlet, gevent.

Warning

When using gevent you might also want to set the _AIRFLOW_PATCH_GEVENT environment variable to "1" to make sure gevent patching is done as early as possible.

See related Issues / PRs for more details:

Type: string
Default: sync
Environment Variable: AIRFLOW__WEBSERVER__WORKER_CLASS

worker_refresh_batch_size¶

Number of workers to refresh at a time. When set to 0, worker refresh is disabled. When nonzero, airflow periodically refreshes webserver workers by bringing up new ones and killing old ones.

Type: string
Default: 1
Environment Variable: AIRFLOW__WEBSERVER__WORKER_REFRESH_BATCH_SIZE

worker_refresh_interval¶

Number of seconds to wait before refreshing a batch of workers.

Type: string
Default: 6000
Environment Variable: AIRFLOW__WEBSERVER__WORKER_REFRESH_INTERVAL

workers¶

Number of workers to run the Gunicorn web server

Type: string
Default: 4
Environment Variable: AIRFLOW__WEBSERVER__WORKERS

x_frame_enabled¶

New in version 1.10.8.

Allow the UI to be rendered in a frame

Type: boolean
Default: True
Environment Variable: AIRFLOW__WEBSERVER__X_FRAME_ENABLED