Customizing Workers¶

Both CeleryExecutor and KubernetesExecutor workers can be highly customized with the workers parameters. For example, to set resources on workers:

workers:
  resources:
    requests:
      cpu: 1
    limits:
      cpu: 1

See workers parameters for a complete list.

One notable exception for KuberntesExecutor is that the default anti-affinity applied to CeleryExecutor workers to spread them across nodes is not applied to KubernetesExecutor workers, as there is no reason to spread out per-task workers.

Custom `pod_template_file`¶

With KubernetesExecutor or CeleryKubernetesExecutor you can also provide a complete pod_template_file to configure Kubernetes workers. This may be useful if you need different configuration between worker types for CeleryKubernetesExecutor or if you need to customize something not possible with workers parameters alone.

As an example, let’s say you want to set priorityClassName on your workers:

Note

The following example is NOT functional, but meant to be illustrative of how you can provide a custom pod_template_file. You’re better off starting with the default pod_template_file instead.

podTemplate: |
  apiVersion: v1
  kind: Pod
  metadata:
    name: dummy-name
    labels:
      tier: airflow
      component: worker
      release: {{ .Release.Name }}
  spec:
    priorityClassName: high-priority
    containers:
      - name: base

Customizing Workers¶

Custom pod_template_file¶

Custom `pod_template_file`¶