airflow.providers.google.cloud.operators.dataplex

This module contains Google Dataplex operators.

Module Contents

Classes

DataplexCreateTaskOperator

Creates a task resource within a lake.

DataplexDeleteTaskOperator

Delete the task resource.

DataplexListTasksOperator

Lists tasks under the given lake.

DataplexGetTaskOperator

Get task resource.

DataplexCreateLakeOperator

Creates a lake resource within a lake.

DataplexDeleteLakeOperator

Delete the lake resource.
class airflow.providers.google.cloud.operators.dataplex.DataplexCreateTaskOperator(project_id, region, lake_id, body, dataplex_task_id, validate_only=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, *args, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Creates a task resource within a lake.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.

  • body (dict[str, Any]) – Required. The Request body contains an instance of Task.

  • dataplex_task_id (str) – Required. Task identifier.

  • validate_only (bool | None) – Optional. Only validate the request, but do not perform mutations. The default is false.

  • api_version (str) – The version of the api that will be requested for example ‘v3’.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

  • asynchronous (bool) – Flag informing should the Dataplex task be created asynchronously. This is useful for long running creating tasks and waiting on them asynchronously using the DataplexTaskSensor

template_fields = ('project_id', 'dataplex_task_id', 'body', 'validate_only', 'impersonation_chain')[source]
template_fields_renderers[source]
execute(context)[source]

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteTaskOperator(project_id, region, lake_id, dataplex_task_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Delete the task resource.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.

  • dataplex_task_id (str) – Required. Task identifier.

  • api_version (str) – The version of the api that will be requested for example ‘v3’.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields = ('project_id', 'dataplex_task_id', 'impersonation_chain')[source]
execute(context)[source]

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.dataplex.DataplexListTasksOperator(project_id, region, lake_id, page_size=None, page_token=None, filter=None, order_by=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Lists tasks under the given lake.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.

  • page_size (int | None) – Optional. Maximum number of tasks to return. The service may return fewer than this value. If unspecified, at most 10 tasks will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000.

  • page_token (str | None) – Optional. Page token received from a previous ListZones call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListZones must match the call that provided the page token.

  • filter (str | None) – Optional. Filter request.

  • order_by (str | None) – Optional. Order by fields for the result.

  • api_version (str) – The version of the api that will be requested for example ‘v3’.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields = ('project_id', 'page_size', 'page_token', 'filter', 'order_by', 'impersonation_chain')[source]
execute(context)[source]

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.dataplex.DataplexGetTaskOperator(project_id, region, lake_id, dataplex_task_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Get task resource.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.

  • dataplex_task_id (str) – Required. Task identifier.

  • api_version (str) – The version of the api that will be requested for example ‘v3’.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields = ('project_id', 'dataplex_task_id', 'impersonation_chain')[source]
execute(context)[source]

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.dataplex.DataplexCreateLakeOperator(project_id, region, lake_id, body, validate_only=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, *args, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Creates a lake resource within a lake.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the lake belongs to.

  • lake_id (str) – Required. Lake identifier.

  • body (dict[str, Any]) – Required. The Request body contains an instance of Lake.

  • validate_only (bool | None) – Optional. Only validate the request, but do not perform mutations. The default is false.

  • api_version (str) – The version of the api that will be requested for example ‘v1’.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

  • asynchronous (bool) – Flag informing should the Dataplex lake be created asynchronously. This is useful for long running creating lakes and waiting on them asynchronously using the DataplexLakeSensor

template_fields = ('project_id', 'lake_id', 'body', 'validate_only', 'impersonation_chain')[source]
template_fields_renderers[source]
execute(context)[source]

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteLakeOperator(project_id, region, lake_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Delete the lake resource.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the lake belongs to.

  • lake_id (str) – Required. Lake identifier.

  • api_version (str) – The version of the api that will be requested for example ‘v1’.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields = ('project_id', 'lake_id', 'impersonation_chain')[source]
execute(context)[source]

This is the main method to derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?