airflow.providers.cncf.kubernetes.operators.pod¶
Executes task in a Kubernetes POD
Module Contents¶
Classes¶
Execute a task in a Kubernetes Pod |
Attributes¶
- exception airflow.providers.cncf.kubernetes.operators.pod.PodReattachFailure[source]¶
Bases:
airflow.exceptions.AirflowExceptionWhen we expect to be able to find a pod but cannot.
- class airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator(*, kubernetes_conn_id=KubernetesHook.default_conn_name, namespace=None, image=None, name=None, random_name_suffix=True, cmds=None, arguments=None, ports=None, volume_mounts=None, volumes=None, env_vars=None, env_from=None, secrets=None, in_cluster=None, cluster_context=None, labels=None, reattach_on_restart=True, startup_timeout_seconds=120, get_logs=True, image_pull_policy=None, annotations=None, container_resources=None, affinity=None, config_file=None, node_selector=None, image_pull_secrets=None, service_account_name=None, is_delete_operator_pod=True, hostnetwork=False, tolerations=None, security_context=None, container_security_context=None, dnspolicy=None, schedulername=None, full_pod_spec=None, init_containers=None, log_events_on_failure=False, do_xcom_push=False, pod_template_file=None, priority_class_name=None, pod_runtime_info_envs=None, termination_grace_period=None, configmaps=None, base_container_name=None, deferrable=False, poll_interval=2, **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorExecute a task in a Kubernetes Pod
See also
For more information on how to use this operator, take a look at the guide: KubernetesPodOperator
Note
If you use Google Kubernetes Engine and Airflow is not running in the same cluster, consider using
GKEStartPodOperator, which simplifies the authorization process.- Parameters
kubernetes_conn_id (str | None) – The kubernetes connection id for the Kubernetes cluster.
namespace (str | None) – the namespace to run within kubernetes.
image (str | None) – Docker image you wish to launch. Defaults to hub.docker.com, but fully qualified URLS will point to custom repositories. (templated)
name (str | None) – name of the pod in which the task will run, will be used (plus a random suffix if random_name_suffix is True) to generate a pod id (DNS-1123 subdomain, containing only [a-z0-9.-]).
random_name_suffix (bool) – if True, will generate a random suffix.
cmds (list[str] | None) – entrypoint of the container. (templated) The docker images’s entrypoint is used if this is not provided.
arguments (list[str] | None) – arguments of the entrypoint. (templated) The docker image’s CMD is used if this is not provided.
ports (list[k8s.V1ContainerPort] | None) – ports for the launched pod.
volume_mounts (list[k8s.V1VolumeMount] | None) – volumeMounts for the launched pod.
volumes (list[k8s.V1Volume] | None) – volumes for the launched pod. Includes ConfigMaps and PersistentVolumes.
env_vars (list[k8s.V1EnvVar] | None) – Environment variables initialized in the container. (templated)
env_from (list[k8s.V1EnvFromSource] | None) – (Optional) List of sources to populate environment variables in the container.
secrets (list[Secret] | None) – Kubernetes secrets to inject in the container. They can be exposed as environment vars or files in a volume.
in_cluster (bool | None) – run kubernetes client with in_cluster configuration.
cluster_context (str | None) – context that points to kubernetes cluster. Ignored when in_cluster is True. If None, current-context is used.
reattach_on_restart (bool) – if the worker dies while the pod is running, reattach and monitor during the next try. If False, always create a new pod for each try.
labels (dict | None) – labels to apply to the Pod. (templated)
startup_timeout_seconds (int) – timeout in seconds to startup the pod.
get_logs (bool) – get the stdout of the base container as logs of the tasks.
image_pull_policy (str | None) – Specify a policy to cache or always pull an image.
annotations (dict | None) – non-identifying metadata you can attach to the Pod. Can be a large range of data, and can include characters that are not permitted by labels.
container_resources (k8s.V1ResourceRequirements | None) – resources for the launched pod. (templated)
affinity (k8s.V1Affinity | None) – affinity scheduling rules for the launched pod.
config_file (str | None) – The path to the Kubernetes config file. (templated) If not specified, default value is
~/.kube/confignode_selector (dict | None) – A dict containing a group of scheduling rules.
image_pull_secrets (list[k8s.V1LocalObjectReference] | None) – Any image pull secrets to be given to the pod. If more than one secret is required, provide a comma separated list: secret_a,secret_b
service_account_name (str | None) – Name of the service account
is_delete_operator_pod (bool) – What to do when the pod reaches its final state, or the execution is interrupted. If True (default), delete the pod; if False, leave the pod.
hostnetwork (bool) – If True enable host networking on the pod.
tolerations (list[k8s.V1Toleration] | None) – A list of kubernetes tolerations.
security_context (dict | None) – security options the pod should run with (PodSecurityContext).
container_security_context (dict | None) – security options the container should run with.
dnspolicy (str | None) – dnspolicy for the pod.
schedulername (str | None) – Specify a schedulername for the pod
full_pod_spec (k8s.V1Pod | None) – The complete podSpec
init_containers (list[k8s.V1Container] | None) – init container for the launched Pod
log_events_on_failure (bool) – Log the pod’s events if a failure occurs
do_xcom_push (bool) – If True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.
pod_template_file (str | None) – path to pod template file (templated)
priority_class_name (str | None) – priority class name for the launched Pod
pod_runtime_info_envs (list[k8s.V1EnvVar] | None) – (Optional) A list of environment variables, to be set in the container.
termination_grace_period (int | None) – Termination grace period if task killed in UI, defaults to kubernetes default
configmaps (list[str] | None) – (Optional) A list of names of config maps from which it collects ConfigMaps to populate the environment variables with. The contents of the target ConfigMap’s Data field will represent the key-value pairs as environment variables. Extends env_from.
base_container_name (str | None) – The name of the base container in the pod. This container’s logs will appear as part of this task’s logs if get_logs is True. Defaults to None. If None, will consult the class variable BASE_CONTAINER_NAME (which defaults to “base”) for the base container name to use.
deferrable (bool) – Run operator in the deferrable mode.
poll_interval (float) – Polling period in seconds to check for the status. Used only in deferrable mode.
- template_fields: Sequence[str] = ('image', 'cmds', 'arguments', 'env_vars', 'labels', 'config_file', 'pod_template_file',...[source]¶
- find_pod(namespace, context, *, exclude_checked=True)[source]¶
Returns an already-running pod for this task instance if one exists.
- execute(context)[source]¶
Based on the deferrable parameter runs the pod asynchronously or synchronously
- invoke_defer_method()[source]¶
Method to easily redefine triggers which are being used in child classes.
- post_complete_action(*, pod, remote_pod, **kwargs)[source]¶
Actions that must be done after operator finishes logic of the deferrable_execution.
- patch_already_checked(pod, *, reraise=True)[source]¶
Add an “already checked” annotation to ensure we don’t reattach on retries
- on_kill()[source]¶
Override this method to clean up subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up, or it will leave ghost processes behind.