CeleryKubernetes Executor

The CeleryKubernetesExecutor allows users to run simultaneously a CeleryExecutor and a KubernetesExecutor. An executor is chosen to run a task based on the task’s queue.

CeleryKubernetesExecutor inherits the scalability of the CeleryExecutor to handle the high load at the peak time and runtime isolation of the KubernetesExecutor.

When to use CeleryKubernetesExecutor

The CeleryKubernetesExecutor should only be used at certain cases, given that it requires setting up the CeleryExecutor and the KubernetesExecutor.

We recommend considering the CeleryKubernetesExecutor when your use case meets:

  1. The number of tasks needed to be scheduled at the peak exceeds the scale that your Kubernetes cluster can comfortably handle

  2. A relative small portion of your tasks requires runtime isolation.

  3. You have plenty of small tasks that can be executed on Celery workers but you also have resource-hungry tasks that will be better to run in predefined environments.

Was this entry helpful?