CeleryKubernetes Executor

The CeleryKubernetesExecutor allows users to run simultaneously CeleryExecutor and a KubernetesExecutor. An executor is chosen to run a task based on the task's queue.

CeleryKubernetesExecutor inherits the scalability of CeleryExecutor to handle the high load at the peak time and runtime isolation of KubernetesExecutor.

When to use CeleryKubernetesExecutor

CeleryKubernetesExecutor should only be used at certain cases, given that it requires setting up CeleryExecutor and KubernetesExecutor.

We recommend considering CeleryKubernetesExecutor when your use case meets:

  1. The number of tasks needed to be scheduled at the peak exceeds the scale that your kubernetes cluster can comfortably handle

  2. A relative small portion of your tasks requires runtime isolation.

  3. You have plenty of small tasks that can be executed on Celery workers but you also have resource-hungry tasks that will be better to run in predefined environments.

Was this entry helpful?