airflow.providers.amazon.aws.operators.eks¶
This module contains Amazon EKS operators.
Module Contents¶
Classes¶
Creates an Amazon EKS Cluster control plane. |
|
Creates an Amazon EKS managed node group for an existing Amazon EKS Cluster. |
|
Creates an AWS Fargate profile for an Amazon EKS cluster. |
|
Deletes the Amazon EKS Cluster control plane and all nodegroups attached to it. |
|
Deletes an Amazon EKS managed node group from an Amazon EKS Cluster. |
|
Deletes an AWS Fargate profile from an Amazon EKS Cluster. |
|
Executes a task in a Kubernetes pod on the specified Amazon EKS Cluster. |
Attributes¶
- airflow.providers.amazon.aws.operators.eks.CAN_NOT_DELETE_MSG = 'A cluster can not be deleted with attached {compute}. Deleting {count} {compute}.'[source]¶
- airflow.providers.amazon.aws.operators.eks.MISSING_ARN_MSG = 'Creating an {compute} requires {requirement} to be passed in.'[source]¶
- airflow.providers.amazon.aws.operators.eks.SUCCESS_MSG = 'No {compute} remain, deleting cluster.'[source]¶
- airflow.providers.amazon.aws.operators.eks.NODEGROUP_FULL_NAME = 'Amazon EKS managed node groups'[source]¶
- class airflow.providers.amazon.aws.operators.eks.EksCreateClusterOperator(cluster_name, cluster_role_arn, resources_vpc_config, compute=DEFAULT_COMPUTE_TYPE, create_cluster_kwargs=None, nodegroup_name=DEFAULT_NODEGROUP_NAME, nodegroup_role_arn=None, create_nodegroup_kwargs=None, fargate_profile_name=DEFAULT_FARGATE_PROFILE_NAME, fargate_pod_execution_role_arn=None, fargate_selectors=None, create_fargate_profile_kwargs=None, wait_for_completion=False, aws_conn_id=DEFAULT_CONN_ID, region=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), waiter_delay=30, waiter_max_attempts=40, **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorCreates an Amazon EKS Cluster control plane.
Optionally, can also create the supporting compute architecture:
- If argument ‘compute’ is provided with a value of ‘nodegroup’, will also
attempt to create an Amazon EKS Managed Nodegroup for the cluster. See
EksCreateNodegroupOperatordocumentation for requirements.
- If argument ‘compute’ is provided with a value of ‘fargate’, will also attempt to create an AWS
Fargate profile for the cluster. See
EksCreateFargateProfileOperatordocumentation for requirements.
See also
For more information on how to use this operator, take a look at the guide: Create an Amazon EKS cluster
- Parameters
cluster_name (str) – The unique name to give to your Amazon EKS Cluster. (templated)
cluster_role_arn (str) – The Amazon Resource Name (ARN) of the IAM role that provides permissions for the Kubernetes control plane to make calls to AWS API operations on your behalf. (templated)
resources_vpc_config (dict) – The VPC configuration used by the cluster control plane. (templated)
compute (str | None) – The type of compute architecture to generate along with the cluster. (templated) Defaults to ‘nodegroup’ to generate an EKS Managed Nodegroup.
create_cluster_kwargs (dict | None) – Optional parameters to pass to the CreateCluster API (templated)
wait_for_completion (bool) – If True, waits for operator to complete. (default: False) (templated)
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
If compute is assigned the value of ‘nodegroup’:
- Parameters
nodegroup_name (str) – REQUIRED The unique name to give your Amazon EKS managed node group. (templated)
nodegroup_role_arn (str | None) – REQUIRED The Amazon Resource Name (ARN) of the IAM role to associate with the Amazon EKS managed node group. (templated)
create_nodegroup_kwargs (dict | None) – Optional parameters to pass to the CreateNodegroup API (templated)
If compute is assigned the value of ‘fargate’:
- Parameters
fargate_profile_name (str) – REQUIRED The unique name to give your AWS Fargate profile. (templated)
fargate_pod_execution_role_arn (str | None) – REQUIRED The Amazon Resource Name (ARN) of the pod execution role to use for pods that match the selectors in the AWS Fargate profile. (templated)
fargate_selectors (list | None) – The selectors to match for pods to use this AWS Fargate profile. (templated)
create_fargate_profile_kwargs (dict | None) – Optional parameters to pass to the CreateFargateProfile API (templated)
waiter_delay (int) – Time (in seconds) to wait between two consecutive calls to check cluster state
waiter_max_attempts (int) – The maximum number of attempts to check cluster state
deferrable (bool) – If True, the operator will wait asynchronously for the job to complete. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
- template_fields: Sequence[str] = ('cluster_name', 'cluster_role_arn', 'resources_vpc_config', 'create_cluster_kwargs', 'compute',...[source]¶
- class airflow.providers.amazon.aws.operators.eks.EksCreateNodegroupOperator(cluster_name, nodegroup_subnets, nodegroup_role_arn, nodegroup_name=DEFAULT_NODEGROUP_NAME, create_nodegroup_kwargs=None, wait_for_completion=False, aws_conn_id=DEFAULT_CONN_ID, region=None, waiter_delay=30, waiter_max_attempts=80, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorCreates an Amazon EKS managed node group for an existing Amazon EKS Cluster.
See also
For more information on how to use this operator, take a look at the guide: Create an Amazon EKS managed node group
- Parameters
cluster_name (str) – The name of the Amazon EKS Cluster to create the managed nodegroup in. (templated)
nodegroup_name (str) – The unique name to give your managed nodegroup. (templated)
nodegroup_subnets (list[str] | str) – The subnets to use for the Auto Scaling group that is created for the managed nodegroup. (templated)
nodegroup_role_arn (str) – The Amazon Resource Name (ARN) of the IAM role to associate with the managed nodegroup. (templated)
create_nodegroup_kwargs (dict | None) – Optional parameters to pass to the Create Nodegroup API (templated)
wait_for_completion (bool) – If True, waits for operator to complete. (default: False) (templated)
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
waiter_delay (int) – Time (in seconds) to wait between two consecutive calls to check nodegroup state
waiter_max_attempts (int) – The maximum number of attempts to check nodegroup state
deferrable (bool) – If True, the operator will wait asynchronously for the nodegroup to be created. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
- template_fields: Sequence[str] = ('cluster_name', 'nodegroup_subnets', 'nodegroup_role_arn', 'nodegroup_name',...[source]¶
- class airflow.providers.amazon.aws.operators.eks.EksCreateFargateProfileOperator(cluster_name, pod_execution_role_arn, selectors, fargate_profile_name=DEFAULT_FARGATE_PROFILE_NAME, create_fargate_profile_kwargs=None, wait_for_completion=False, aws_conn_id=DEFAULT_CONN_ID, region=None, waiter_delay=10, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorCreates an AWS Fargate profile for an Amazon EKS cluster.
See also
For more information on how to use this operator, take a look at the guide: Create an AWS Fargate Profile
- Parameters
cluster_name (str) – The name of the Amazon EKS cluster to apply the AWS Fargate profile to. (templated)
pod_execution_role_arn (str) – The Amazon Resource Name (ARN) of the pod execution role to use for pods that match the selectors in the AWS Fargate profile. (templated)
selectors (list) – The selectors to match for pods to use this AWS Fargate profile. (templated)
fargate_profile_name (str) – The unique name to give your AWS Fargate profile. (templated)
create_fargate_profile_kwargs (dict | None) – Optional parameters to pass to the CreateFargate Profile API (templated)
wait_for_completion (bool) – If True, waits for operator to complete. (default: False) (templated)
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
waiter_delay (int) – Time (in seconds) to wait between two consecutive calls to check profile status
waiter_max_attempts (int) – The maximum number of attempts to check the status of the profile.
deferrable (bool) – If True, the operator will wait asynchronously for the profile to be created. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
- template_fields: Sequence[str] = ('cluster_name', 'pod_execution_role_arn', 'selectors', 'fargate_profile_name',...[source]¶
- class airflow.providers.amazon.aws.operators.eks.EksDeleteClusterOperator(cluster_name, force_delete_compute=False, wait_for_completion=False, aws_conn_id=DEFAULT_CONN_ID, region=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), waiter_delay=30, waiter_max_attempts=40, **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorDeletes the Amazon EKS Cluster control plane and all nodegroups attached to it.
See also
For more information on how to use this operator, take a look at the guide: Delete an Amazon EKS Cluster
- Parameters
cluster_name (str) – The name of the Amazon EKS Cluster to delete. (templated)
force_delete_compute (bool) – If True, will delete any attached resources. (templated) Defaults to False.
wait_for_completion (bool) – If True, waits for operator to complete. (default: False) (templated)
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
waiter_delay (int) – Time (in seconds) to wait between two consecutive calls to check cluster state
waiter_max_attempts (int) – The maximum number of attempts to check cluster state
deferrable (bool) – If True, the operator will wait asynchronously for the cluster to be deleted. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
- template_fields: Sequence[str] = ('cluster_name', 'force_delete_compute', 'wait_for_completion', 'aws_conn_id', 'region')[source]¶
- execute(context)[source]¶
This is the main method to derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
- delete_any_nodegroups(eks_hook)[source]¶
Deletes all Amazon EKS managed node groups for a provided Amazon EKS Cluster.
Amazon EKS managed node groups can be deleted in parallel, so we can send all delete commands in bulk and move on once the count of nodegroups is zero.
- class airflow.providers.amazon.aws.operators.eks.EksDeleteNodegroupOperator(cluster_name, nodegroup_name, wait_for_completion=False, aws_conn_id=DEFAULT_CONN_ID, region=None, waiter_delay=30, waiter_max_attempts=40, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorDeletes an Amazon EKS managed node group from an Amazon EKS Cluster.
See also
For more information on how to use this operator, take a look at the guide: Delete an Amazon EKS managed node group
- Parameters
cluster_name (str) – The name of the Amazon EKS Cluster associated with your nodegroup. (templated)
nodegroup_name (str) – The name of the nodegroup to delete. (templated)
wait_for_completion (bool) – If True, waits for operator to complete. (default: False) (templated)
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
waiter_delay (int) – Time (in seconds) to wait between two consecutive calls to check nodegroup state
waiter_max_attempts (int) – The maximum number of attempts to check nodegroup state
deferrable (bool) – If True, the operator will wait asynchronously for the nodegroup to be deleted. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
- template_fields: Sequence[str] = ('cluster_name', 'nodegroup_name', 'wait_for_completion', 'aws_conn_id', 'region')[source]¶
- class airflow.providers.amazon.aws.operators.eks.EksDeleteFargateProfileOperator(cluster_name, fargate_profile_name, wait_for_completion=False, aws_conn_id=DEFAULT_CONN_ID, region=None, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorDeletes an AWS Fargate profile from an Amazon EKS Cluster.
See also
For more information on how to use this operator, take a look at the guide: Delete an AWS Fargate Profile
- Parameters
cluster_name (str) – The name of the Amazon EKS cluster associated with your Fargate profile. (templated)
fargate_profile_name (str) – The name of the AWS Fargate profile to delete. (templated)
wait_for_completion (bool) – If True, waits for operator to complete. (default: False) (templated)
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
waiter_delay (int) – Time (in seconds) to wait between two consecutive calls to check profile status
waiter_max_attempts (int) – The maximum number of attempts to check the status of the profile.
deferrable (bool) – If True, the operator will wait asynchronously for the profile to be deleted. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
- template_fields: Sequence[str] = ('cluster_name', 'fargate_profile_name', 'wait_for_completion', 'aws_conn_id', 'region')[source]¶
- class airflow.providers.amazon.aws.operators.eks.EksPodOperator(cluster_name, in_cluster=False, namespace=DEFAULT_NAMESPACE_NAME, pod_context=None, pod_name=None, pod_username=None, aws_conn_id=DEFAULT_CONN_ID, region=None, on_finish_action=None, is_delete_operator_pod=None, **kwargs)[source]¶
Bases:
airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperatorExecutes a task in a Kubernetes pod on the specified Amazon EKS Cluster.
See also
For more information on how to use this operator, take a look at the guide: Perform a Task on an Amazon EKS Cluster
- Parameters
cluster_name (str) – The name of the Amazon EKS Cluster to execute the task on. (templated)
cluster_role_arn – The Amazon Resource Name (ARN) of the IAM role that provides permissions for the Kubernetes control plane to make calls to AWS API operations on your behalf. (templated)
in_cluster (bool) – If True, look for config inside the cluster; if False look for a local file path.
namespace (str) – The namespace in which to execute the pod. (templated)
pod_name (str | None) – The unique name to give the pod. (templated)
aws_profile – The named profile containing the credentials for the AWS CLI tool to use.
region (str | None) – Which AWS region the connection should use. (templated) If this is None or empty then the default boto3 behaviour is used.
aws_conn_id (str) – The Airflow connection used for AWS credentials. (templated) If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then the default boto3 configuration would be used (and must be maintained on each worker node).
on_finish_action (str | None) – What to do when the pod reaches its final state, or the execution is interrupted. If “delete_pod”, the pod will be deleted regardless it’s state; if “delete_succeeded_pod”, only succeeded pod will be deleted. You can set to “keep_pod” to keep the pod. Current default is keep_pod, but this will be changed in the next major release of this provider.
is_delete_operator_pod (bool | None) – What to do when the pod reaches its final state, or the execution is interrupted. If True, delete the pod; if False, leave the pod. Current default is False, but this will be changed in the next major release of this provider. Deprecated - use on_finish_action instead.