airflow.executors.celery_executor

Module Contents

airflow.executors.celery_executor.CELERY_FETCH_ERR_MSG_HEADER = Error fetching Celery task state[source]
airflow.executors.celery_executor.CELERY_SEND_ERR_MSG_HEADER = Error sending Celery task[source]
airflow.executors.celery_executor.OPERATION_TIMEOUT[source]

To start the celery worker, run the command: airflow worker

airflow.executors.celery_executor.celery_configuration[source]
airflow.executors.celery_executor.app[source]
airflow.executors.celery_executor.execute_command(command_to_exec)[source]
class airflow.executors.celery_executor.ExceptionWithTraceback(exception, exception_traceback)[source]

Bases: object

Wrapper class used to propagate exceptions to parent processes from subprocesses.

Parameters

  • exception (Exception) – The exception to wrap

  • exception_traceback (str) – The stacktrace to wrap

airflow.executors.celery_executor.fetch_celery_task_state(celery_task)[source]
Fetch and return the state of the given celery task. The scope of this function is
global so that it can be called by subprocesses in the pool.
Parameters

celery_task (tuple(str, celery.result.AsyncResult)) – a tuple of the Celery task key and the async Celery object used to fetch the task’s state

Returns

a tuple of the Celery task key and the Celery state of the task

Return type

tuple[str, str]

airflow.executors.celery_executor.send_task_to_executor(task_tuple)[source]
class airflow.executors.celery_executor.CeleryExecutor[source]

Bases: airflow.executors.base_executor.BaseExecutor

CeleryExecutor is recommended for production use of Airflow. It allows distributing the execution of task instances to multiple worker nodes.

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

start(self)[source]
_num_tasks_per_send_process(self, to_send_count)[source]

How many Celery tasks should each worker process send.

Returns

Number of tasks that should be sent per process

Return type

int

_num_tasks_per_fetch_process(self)[source]

How many Celery tasks should be sent to each worker process.

Returns

Number of tasks that should be used per process

Return type

int

trigger_tasks(self, open_slots)[source]

Overwrite trigger_tasks function from BaseExecutor

Parameters

open_slots – Number of open slots

Returns

sync(self)[source]
end(self, synchronous=False)[source]

Was this entry helpful?