`airflow.providers.celery.executors.celery_executor`¶

CeleryExecutor.

Module Contents¶

Classes¶

CeleryExecutor

CeleryExecutor is recommended for production use of Airflow.

Functions¶

__getattr__(name)

Attributes¶

`log`
`CELERY_SEND_ERR_MSG_HEADER`
`TaskInstanceInCelery`
`ARG_BROKER_API`
`ARG_FLOWER_HOSTNAME`
`ARG_FLOWER_PORT`
`ARG_FLOWER_CONF`
`ARG_FLOWER_URL_PREFIX`
`ARG_FLOWER_BASIC_AUTH`
`ARG_AUTOSCALE`
`ARG_QUEUES`
`ARG_CONCURRENCY`
`ARG_CELERY_HOSTNAME`
`ARG_UMASK`
`ARG_WITHOUT_MINGLE`
`ARG_WITHOUT_GOSSIP`
`CELERY_CLI_COMMAND_PATH`
`CELERY_COMMANDS`

airflow.providers.celery.executors.celery_executor.log[source]¶

airflow.providers.celery.executors.celery_executor.CELERY_SEND_ERR_MSG_HEADER = 'Error sending Celery task'[source]¶

airflow.providers.celery.executors.celery_executor.TaskInstanceInCelery[source]¶

airflow.providers.celery.executors.celery_executor.__getattr__(name)[source]¶

airflow.providers.celery.executors.celery_executor.ARG_BROKER_API[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_HOSTNAME[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_PORT[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_CONF[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_URL_PREFIX[source]¶

airflow.providers.celery.executors.celery_executor.ARG_FLOWER_BASIC_AUTH[source]¶

airflow.providers.celery.executors.celery_executor.ARG_AUTOSCALE[source]¶

airflow.providers.celery.executors.celery_executor.ARG_QUEUES[source]¶

airflow.providers.celery.executors.celery_executor.ARG_CONCURRENCY[source]¶

airflow.providers.celery.executors.celery_executor.ARG_CELERY_HOSTNAME[source]¶

airflow.providers.celery.executors.celery_executor.ARG_UMASK[source]¶

airflow.providers.celery.executors.celery_executor.ARG_WITHOUT_MINGLE[source]¶

airflow.providers.celery.executors.celery_executor.ARG_WITHOUT_GOSSIP[source]¶

airflow.providers.celery.executors.celery_executor.CELERY_CLI_COMMAND_PATH[source]¶

airflow.providers.celery.executors.celery_executor.CELERY_COMMANDS = ()[source]¶

class airflow.providers.celery.executors.celery_executor.CeleryExecutor[source]¶

Bases: airflow.executors.base_executor.BaseExecutor

CeleryExecutor is recommended for production use of Airflow.

It allows distributing the execution of task instances to multiple worker nodes.

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

supports_ad_hoc_ti_run: bool = True[source]¶

supports_sentry: bool = True[source]¶

start()[source]¶

Executors may need to get things started.

sync()[source]¶

Sync will get called periodically by the heartbeat method.

Executors should override this to perform gather statuses.

debug_dump()[source]¶

Debug dump; called in response to SIGUSR2 by the scheduler.

update_all_task_states()[source]¶

Update states of the tasks.

change_state(key, state, info=None, remove_running=True)[source]¶

Change state of the task.

Parameters

key (airflow.models.taskinstancekey.TaskInstanceKey) – Unique key for the task instance
state (airflow.utils.state.TaskInstanceState) – State to set for the task.
info – Executor information for the task instance
remove_running – Whether or not to remove the TI key from running set

update_task_state(key, state, info)[source]¶

Update state of a single task.

end(synchronous=False)[source]¶

Wait synchronously for the previously submitted job to complete.

terminate()[source]¶: Get called when the daemon receives a SIGTERM.

try_adopt_task_instances(tis)[source]¶

Try to adopt running task instances that have been abandoned by a SchedulerJob dying.

Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)

Returns: any TaskInstances that were unable to be adopted
Return type: Sequence[airflow.models.taskinstance.TaskInstance]

cleanup_stuck_queued_tasks(tis)[source]¶

Handle remnants of tasks that were failed because they were stuck in queued.

Tasks can get stuck in queued. If such a task is detected, it will be marked as UP_FOR_RETRY if the task instance has remaining retries or marked as FAILED if it doesn’t.

Parameters: tis (list[airflow.models.taskinstance.TaskInstance]) – List of Task Instances to clean up
Returns: List of readable task instances for a warning message
Return type: list[str]

static get_cli_commands()[source]¶

Vends CLI commands to be included in Airflow CLI.

Override this method to expose commands via Airflow CLI to manage this executor. This can be commands to setup/teardown the executor, inspect state, etc. Make sure to choose unique names for those commands, to avoid collisions.

airflow.providers.celery.executors.celery_executor¶