Configuration Reference

This page contains the list of all available Airflow configurations for the apache-airflow-providers-edge3 provider that can be set in the airflow.cfg file or using environment variables.

Note

For more information see Setting Configuration Options.

[edge]

This section only applies if you are using the EdgeExecutor in [core] section above

api_enabled

Flag if the plugin endpoint is enabled to serve Edge Workers.

Type:

boolean

Default:

False

Environment Variable:

AIRFLOW__EDGE__API_ENABLED

Example:

True

api_url

URL endpoint on which the Airflow code edge API is accessible from edge worker.

Type:

string

Default:

None

Environment Variable:

AIRFLOW__EDGE__API_URL

Example:

https://airflow.hosting.org/edge_worker/v1/rpcapi

automatic_maintenance_exit

Added in version 3.6.0.

Enable automatic exit from maintenance mode for edge workers based on reported system info status.

When configured, edge workers will automatically exit maintenance mode that was set by automatic maintenance mode when the reported system info status indicates that the worker is healthy (again). So in conjunction with a extended_system_info_function that reports system status, this can be used to automatically bring workers back when they are healthy.

Possible options are “Warning”, “Info” and “Off”. When set to “Warning”, workers will exit maintenance mode when the system info function reports a status of “Warning” or “Info”. When set to “Info”, workers will exit maintenance mode when the system info function reports a status of “Info”. When set to “Off”, automatic maintenance exit is disabled and workers will not automatically exit maintenance mode based on system info status.

Type:

string

Default:

Off

Environment Variable:

AIRFLOW__EDGE__AUTOMATIC_MAINTENANCE_EXIT

Example:

Info

automatic_maintenance_on

Added in version 3.6.0.

Enable automatic maintenance mode for edge workers based on reported system info status.

When configured, edge workers will automatically enter maintenance mode when the reported system info status indicates a problem on the worker. So in conjunction with a extended_system_info_function that reports system status, this can be used to automatically take workers out when they are unhealthy.

Possible options are “Error”, “Warning” and “Off”. When set to “Error”, workers will enter maintenance mode when the system info function reports a status of “Error”. When set to “Warning”, workers will enter maintenance mode when the system info function reports a status of “Error” or “Warning”.

When set to “Off”, automatic maintenance mode is disabled and workers will not automatically enter maintenance mode based on system info status.

Type:

string

Default:

Off

Environment Variable:

AIRFLOW__EDGE__AUTOMATIC_MAINTENANCE_ON

Example:

Error

drain_kill_grace_sec

Added in version 3.6.0.

Grace period in seconds after SIGTERM before the worker escalates to SIGKILL and exits. Only used when drain_timeout_sec is greater than 0.

Type:

integer

Default:

30

Environment Variable:

AIRFLOW__EDGE__DRAIN_KILL_GRACE_SEC

Example:

30

drain_timeout_sec

Added in version 3.6.0.

Maximum seconds the worker waits for running jobs to finish after entering drain (shutdown requested via SIGINT / SIGTERM / SHUTDOWN_REQUEST / version mismatch). Once this timeout elapses the worker sends SIGTERM to each running job, then SIGKILL after drain_kill_grace_sec, and exits regardless of remaining jobs. Set to 0 (the default) to wait indefinitely and preserve legacy behavior.

Type:

integer

Default:

0

Environment Variable:

AIRFLOW__EDGE__DRAIN_TIMEOUT_SEC

Example:

3600

extended_system_info_function

Added in version 3.5.0.

The function to call to get extended system information for the worker.

The function must be async and return a dict[str, str | int | float | datetime]. The information will be sent to the central site with each heartbeat and can be used for monitoring and debugging purposes. All int and float values will also be published to metric collection systems like statsd or otel.

Function must be provided as a string with the full path to the function. See https://github.com/apache/airflow/blob/main/providers/edge3/src/airflow/providers/edge3/cli/example_extended_sysinfo.py for an example implementation.

Type:

string

Default:

None

Environment Variable:

AIRFLOW__EDGE__EXTENDED_SYSTEM_INFO_FUNCTION

Example:

airflow.providers.edge3.cli.example_extended_sysinfo.get_example_extended_sysinfo

heartbeat_interval

Edge Worker continuously reports status to the central site. This parameter defines how often a status with heartbeat should be sent. During heartbeat status is reported as well as it is checked if a running task is to be terminated.

Type:

integer

Default:

30

Environment Variable:

AIRFLOW__EDGE__HEARTBEAT_INTERVAL

Example:

10

job_fail_purge

Minutes after which failed jobs for EdgeExecutor are purged from database

Type:

integer

Default:

60

Environment Variable:

AIRFLOW__EDGE__JOB_FAIL_PURGE

job_poll_interval

Edge Worker currently polls for new jobs via HTTP. This parameter defines the number of seconds it should sleep between polls for new jobs. Job polling only happens if the Edge Worker seeks for new work. Not if busy.

Type:

integer

Default:

5

Environment Variable:

AIRFLOW__EDGE__JOB_POLL_INTERVAL

Example:

5

job_success_purge

Minutes after which successful jobs for EdgeExecutor are purged from database

Type:

integer

Default:

5

Environment Variable:

AIRFLOW__EDGE__JOB_SUCCESS_PURGE

minimum_acceptable_core_version_for_workers

Added in version 3.6.0.

The minimum acceptable version of the central Airflow site and edge worker to work with.

This is used to make sure that the central Airflow site and edge workers are compatible with each other. When an edge worker connects to the central site, it reports its installed airflow core version. If the version is lower than the minimum acceptable version, the central site will notify the worker during heartbeat and raise an alert. The worker will gracefully terminate and drain jobs and shut down.

If you know your workers are still compatible with older versions of the central site, you can set this to a lower version to avoid having to update all workers at the same time as the central site. Default is to force matching versions, so relaxing is optional to be configured.

If the airflow core version between edge worker and central site is not matching, the worker will report warning status.

Type:

string

Default:

None

Environment Variable:

AIRFLOW__EDGE__MINIMUM_ACCEPTABLE_CORE_VERSION_FOR_WORKERS

Example:

3.2.1

minimum_acceptable_edge_version_for_workers

Added in version 3.6.0.

The minimum acceptable version of the edge worker to work with the central site.

This is used to make sure that the central Airflow site and edge workers are compatible with each other. When an edge worker connects to the central site, it reports its edge version. If the version is lower than the minimum acceptable version, the central site will notify the worker during heartbeat and raise an alert. The worker will gracefully terminate and drain jobs and shut down.

If you know your edge workers are still compatible with older versions of the central site, you can set this to a lower version to avoid having to update all workers at the same time as the central site. Default is to force matching versions, so relaxing is optional to be configured.

If the edge version between edge worker and central site is not matching, the worker will report warning status.

Type:

string

Default:

None

Environment Variable:

AIRFLOW__EDGE__MINIMUM_ACCEPTABLE_EDGE_VERSION_FOR_WORKERS

Example:

3.5.0

push_log_chunk_size

Edge Worker uploads log files in chunks. If the log file part which is uploaded exceeds the chunk size it creates a new request. The application gateway can limit the max body size see: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size A HTTP 413 issue can point to this value to fix the issue. This value must be defined in Bytes.

Type:

integer

Default:

524288

Environment Variable:

AIRFLOW__EDGE__PUSH_LOG_CHUNK_SIZE

push_logs

Added in version 1.5.0.

Flag to enable or disable pushing of log files from edge worker to the central site. When enabled, edge workers will upload task log files in chunks to the central Airflow site. When disabled, logs will only be available locally on the edge worker.

Type:

boolean

Default:

True

Environment Variable:

AIRFLOW__EDGE__PUSH_LOGS

Example:

True

worker_concurrency

The concurrency defines the default max parallel running task instances and can also be set during start of worker with the airflow edge worker command parameter. The size of the workers and the resources must support the nature of your tasks. The parameter works together with the concurrency_slots parameter of a task.

Type:

integer

Default:

8

Environment Variable:

AIRFLOW__EDGE__WORKER_CONCURRENCY

worker_umask

The default umask to use for edge worker when run in daemon mode

This controls the file-creation mode mask which determines the initial value of file permission bits for newly created files.

This value is treated as an octal-integer.

Type:

string

Default:

None

Environment Variable:

AIRFLOW__EDGE__WORKER_UMASK

Was this entry helpful?