Customizing Workers¶
Both CeleryExecutor
and KubernetesExecutor
workers can be highly customized with the workers parameters.
For example, to set resources on workers:
workers:
resources:
requests:
cpu: 1
limits:
cpu: 1
See workers parameters for a complete list.
One notable exception for KubernetesExecutor
is that the default anti-affinity applied to CeleryExecutor
workers to spread them across nodes
is not applied to KubernetesExecutor
workers, as there is no reason to spread out per-task workers.
Custom pod_template_file
¶
With KubernetesExecutor
or CeleryKubernetesExecutor
you can also provide a complete pod_template_file
to configure Kubernetes workers.
This may be useful if you need different configuration between worker types for CeleryKubernetesExecutor
or if you need to customize something not possible with workers parameters alone.
As an example, let’s say you want to set priorityClassName
on your workers:
Note
The following example is NOT functional, but meant to be illustrative of how you can provide a custom pod_template_file
.
You’re better off starting with the default pod_template_file instead.
podTemplate: |
apiVersion: v1
kind: Pod
metadata:
name: placeholder-name
labels:
tier: airflow
component: worker
release: {{ .Release.Name }}
spec:
priorityClassName: high-priority
containers:
- name: base