airflow.providers.celery.executors.celery_executor¶

CeleryExecutor.

Attributes¶

`log`
`CELERY_SEND_ERR_MSG_HEADER`
`ARG_BROKER_API`
`ARG_FLOWER_HOSTNAME`
`ARG_FLOWER_PORT`
`ARG_FLOWER_CONF`
`ARG_FLOWER_URL_PREFIX`
`ARG_FLOWER_BASIC_AUTH`
`ARG_AUTOSCALE`
`ARG_QUEUES`
`ARG_CONCURRENCY`
`ARG_CELERY_HOSTNAME`
`ARG_UMASK`
`ARG_WITHOUT_MINGLE`
`ARG_WITHOUT_GOSSIP`
`CELERY_CLI_COMMAND_PATH`
`CELERY_COMMANDS`

Classes¶

CeleryExecutor

CeleryExecutor is recommended for production use of Airflow.

Functions¶

__getattr__(name)

Module Contents¶

airflow.providers.celery.executors.celery_executor.log[source]¶

airflow.providers.celery.executors.celery_executor.CELERY_SEND_ERR_MSG_HEADER = 'Error sending Celery task'[source]¶

airflow.providers.celery.executors.celery_executor.__getattr__(name)[source]¶

airflow.providers.celery.executors.celery_executor.ARG_BROKER_API[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_HOSTNAME[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_PORT[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_CONF[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_URL_PREFIX[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_BASIC_AUTH[source]¶

airflow.providers.celery.executors.celery_executor.ARG_AUTOSCALE[source]¶

airflow.providers.celery.executors.celery_executor.ARG_QUEUES[source]¶

airflow.providers.celery.executors.celery_executor.ARG_CONCURRENCY[source]¶

airflow.providers.celery.executors.celery_executor.ARG_CELERY_HOSTNAME[source]¶

airflow.providers.celery.executors.celery_executor.ARG_UMASK[source]¶

airflow.providers.celery.executors.celery_executor.ARG_WITHOUT_MINGLE[source]¶

airflow.providers.celery.executors.celery_executor.ARG_WITHOUT_GOSSIP[source]¶

airflow.providers.celery.executors.celery_executor.CELERY_CLI_COMMAND_PATH = 'airflow.providers.celery.cli.celery_command'[source]¶

airflow.providers.celery.executors.celery_executor.CELERY_COMMANDS[source]¶

class airflow.providers.celery.executors.celery_executor.CeleryExecutor[source]¶

Bases: airflow.executors.base_executor.BaseExecutor

CeleryExecutor is recommended for production use of Airflow.

It allows distributing the execution of task instances to multiple worker nodes.

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

supports_ad_hoc_ti_run: bool = True[source]¶

supports_sentry: bool = True[source]¶

queued_tasks: dict[airflow.models.taskinstancekey.TaskInstanceKey, airflow.executors.workloads.All][source]¶

bulk_state_fetcher[source]¶

tasks[source]¶

task_publish_retries: collections.Counter[airflow.models.taskinstancekey.TaskInstanceKey][source]¶

task_publish_max_retries[source]¶

start()[source]¶

Executors may need to get things started.

sync()[source]¶

Sync will get called periodically by the heartbeat method.

Executors should override this to perform gather statuses.

debug_dump()[source]¶

Debug dump; called in response to SIGUSR2 by the scheduler.

update_all_task_states()[source]¶

Update states of the tasks.

change_state(key, state, info=None, remove_running=True)[source]¶

Change state of the task.

Parameters:

key (airflow.models.taskinstancekey.TaskInstanceKey) – Unique key for the task instance
state (airflow.utils.state.TaskInstanceState) – State to set for the task.
info – Executor information for the task instance
remove_running – Whether or not to remove the TI key from running set

update_task_state(key, state, info)[source]¶

Update state of a single task.

end(synchronous=False)[source]¶

Wait synchronously for the previously submitted job to complete.

terminate()[source]¶: Get called when the daemon receives a SIGTERM.

try_adopt_task_instances(tis)[source]¶

Try to adopt running task instances that have been abandoned by a SchedulerJob dying.

Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)

Returns:: any TaskInstances that were unable to be adopted
Return type:: collections.abc.Sequence[airflow.models.taskinstance.TaskInstance]

cleanup_stuck_queued_tasks(tis)[source]¶

Remove tasks stuck in queued from executor and fail them.

This method is deprecated. Use cleanup_tasks_stuck_in_queued instead.

revoke_task(*, ti)[source]¶

Attempt to remove task from executor.

It should attempt to ensure that the task is no longer running on the worker, and ensure that it is cleared out from internal data structures.

It should not change the state of the task in airflow, or add any events to the event buffer.

It should not raise any error.

Parameters:: ti (airflow.models.taskinstance.TaskInstance) – Task instance to remove

static get_cli_commands()[source]¶

Vends CLI commands to be included in Airflow CLI.

Override this method to expose commands via Airflow CLI to manage this executor. This can be commands to setup/teardown the executor, inspect state, etc. Make sure to choose unique names for those commands, to avoid collisions.

queue_workload(workload, session)[source]¶