airflow.providers.google.cloud.operators.dataplex
¶
This module contains Google Dataplex operators.
Module Contents¶
Classes¶
Creates a task resource within a lake. |
|
Delete the task resource. |
|
Lists tasks under the given lake. |
|
Get task resource. |
|
Creates a lake resource within a lake. |
|
Delete the lake resource. |
|
Creates a DataScan resource. |
|
Gets a DataScan resource. |
|
Deletes a DataScan resource. |
|
Runs an on-demand execution of a DataScan. |
|
Gets a Data Scan Job resource. |
|
Creates a DataScan Data Profile resource. |
|
Gets a DataScan DataProfile resource. |
|
Deletes a DataScan DataProfile resource. |
|
Runs an on-demand execution of a DataScan Data Profile Scan. |
|
Gets a DataScan Data Profile Job resource. |
|
Creates a Zone resource within a Lake. |
|
Deletes a Zone resource. All assets within a zone must be deleted before the zone can be deleted. |
|
Creates an Asset resource. |
|
Deletes an asset resource. |
- class airflow.providers.google.cloud.operators.dataplex.DataplexCreateTaskOperator(project_id, region, lake_id, body, dataplex_task_id, validate_only=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a task resource within a lake.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
body (dict[str, Any]) – Required. The Request body contains an instance of Task.
dataplex_task_id (str) – Required. Task identifier.
validate_only (bool | None) – Optional. Only validate the request, but do not perform mutations. The default is false.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
asynchronous (bool) – Flag informing should the Dataplex task be created asynchronously. This is useful for long running creating tasks and waiting on them asynchronously using the DataplexTaskSensor
- class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteTaskOperator(project_id, region, lake_id, dataplex_task_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Delete the task resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
dataplex_task_id (str) – Required. Task identifier.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataplex.DataplexListTasksOperator(project_id, region, lake_id, page_size=None, page_token=None, filter=None, order_by=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Lists tasks under the given lake.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
page_size (int | None) – Optional. Maximum number of tasks to return. The service may return fewer than this value. If unspecified, at most 10 tasks will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000.
page_token (str | None) – Optional. Page token received from a previous ListZones call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListZones must match the call that provided the page token.
filter (str | None) – Optional. Filter request.
order_by (str | None) – Optional. Order by fields for the result.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataplex.DataplexGetTaskOperator(project_id, region, lake_id, dataplex_task_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Get task resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
dataplex_task_id (str) – Required. Task identifier.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataplex.DataplexCreateLakeOperator(project_id, region, lake_id, body, validate_only=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a lake resource within a lake.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. Lake identifier.
body (dict[str, Any]) – Required. The Request body contains an instance of Lake.
validate_only (bool | None) – Optional. Only validate the request, but do not perform mutations. The default is false.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
asynchronous (bool) – Flag informing should the Dataplex lake be created asynchronously. This is useful for long-running creating lakes.
- class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteLakeOperator(project_id, region, lake_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Delete the lake resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. Lake identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataplex.DataplexCreateOrUpdateDataQualityScanOperator(project_id, region, data_scan_id, body, api_version='v1', retry=DEFAULT, timeout=None, update_mask=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a DataScan resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
body (dict[str, Any] | google.cloud.dataplex_v1.types.DataScan) – Required. The Request body contains an instance of DataScan.
data_scan_id (str) – Required. Data Quality scan identifier.
update_mask (dict | google.protobuf.field_mask_pb2.FieldMask | None) – Mask of fields to update.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
Dataplex data scan id
- class airflow.providers.google.cloud.operators.dataplex.DataplexGetDataQualityScanOperator(project_id, region, data_scan_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Gets a DataScan resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
Dataplex data scan
- class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteDataQualityScanOperator(project_id, region, data_scan_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Deletes a DataScan resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
None
- class airflow.providers.google.cloud.operators.dataplex.DataplexRunDataQualityScanOperator(project_id, region, data_scan_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, fail_on_dq_failure=False, result_timeout=60.0 * 10, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), polling_interval_seconds=10, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Runs an on-demand execution of a DataScan.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
asynchronous (bool) – Flag informing that the Dataplex job should be run asynchronously. This is useful for submitting long-running jobs and waiting on them asynchronously using the DataplexDataQualityJobStatusSensor
fail_on_dq_failure (bool) – If set to true and not all Data Quality scan rules have been passed, an exception is thrown. If set to false and not all Data Quality scan rules have been passed, execution will finish with success.
result_timeout (float) – Value in seconds for which operator will wait for the Data Quality scan result when the flag asynchronous = False. Throws exception if there is no result found after specified amount of seconds.
polling_interval_seconds (int) – time in seconds between polling for job completion. The value is considered only when running in deferrable mode. Must be greater than 0.
deferrable (bool) – Run operator in the deferrable mode.
- Returns
Dataplex Data Quality scan job id.
- class airflow.providers.google.cloud.operators.dataplex.DataplexGetDataQualityScanResultOperator(project_id, region, data_scan_id, job_id=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, fail_on_dq_failure=False, wait_for_results=True, result_timeout=60.0 * 10, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), polling_interval_seconds=10, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Gets a Data Scan Job resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
job_id (str | None) – Optional. Data Quality scan job identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if
retry
is specified, the timeout applies to each individual attempt.metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
fail_on_dq_failure (bool) – If set to true and not all Data Quality scan rules have been passed, an exception is thrown. If set to false and not all Data Quality scan rules have been passed, execution will finish with success.
wait_for_results (bool) – Flag indicating whether to wait for the result of a job execution or to return the job in its current state.
result_timeout (float) – Value in seconds for which operator will wait for the Data Quality scan result when the flag wait_for_results = True. Throws exception if there is no result found after specified amount of seconds.
polling_interval_seconds (int) – time in seconds between polling for job completion. The value is considered only when running in deferrable mode. Must be greater than 0.
deferrable (bool) – Run operator in the deferrable mode.
- Returns
Dict representing DataScanJob. When the job completes with a successful status, information about the Data Quality result is available.
- class airflow.providers.google.cloud.operators.dataplex.DataplexCreateOrUpdateDataProfileScanOperator(project_id, region, data_scan_id, body, api_version='v1', retry=DEFAULT, timeout=None, update_mask=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a DataScan Data Profile resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
body (dict[str, Any] | google.cloud.dataplex_v1.types.DataScan) – Required. The Request body contains an instance of DataScan.
data_scan_id (str) – Required. Data Profile scan identifier.
update_mask (dict | google.protobuf.field_mask_pb2.FieldMask | None) – Mask of fields to update.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
Dataplex data profile id
- class airflow.providers.google.cloud.operators.dataplex.DataplexGetDataProfileScanOperator(project_id, region, data_scan_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Gets a DataScan DataProfile resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Profile scan identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
Dataplex data profile
- class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteDataProfileScanOperator(project_id, region, data_scan_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Deletes a DataScan DataProfile resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Profile scan identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
None
- class airflow.providers.google.cloud.operators.dataplex.DataplexRunDataProfileScanOperator(project_id, region, data_scan_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, result_timeout=60.0 * 10, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), polling_interval_seconds=10, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Runs an on-demand execution of a DataScan Data Profile Scan.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Profile scan identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
asynchronous (bool) – Flag informing that the Dataplex job should be run asynchronously. This is useful for submitting long-running jobs and waiting on them asynchronously using the DataplexDataProfileJobStatusSensor
result_timeout (float) – Value in seconds for which operator will wait for the Data Profile scan result when the flag asynchronous = False. Throws exception if there is no result found after specified amount of seconds.
polling_interval_seconds (int) – time in seconds between polling for job completion. The value is considered only when running in deferrable mode. Must be greater than 0.
deferrable (bool) – Run operator in the deferrable mode.
- Returns
Dataplex Data Profile scan job id.
- class airflow.providers.google.cloud.operators.dataplex.DataplexGetDataProfileScanResultOperator(project_id, region, data_scan_id, job_id=None, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, wait_for_results=True, result_timeout=60.0 * 10, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Gets a DataScan Data Profile Job resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Profile scan identifier.
job_id (str | None) – Optional. Data Profile scan job identifier.
api_version (str) – The version of the api that will be requested for example ‘v1’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if
retry
is specified, the timeout applies to each individual attempt.metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
wait_for_results (bool) – Flag indicating whether to wait for the result of a job execution or to return the job in its current state.
result_timeout (float) – Value in seconds for which operator will wait for the Data Profile scan result when the flag wait_for_results = True. Throws exception if there is no result found after specified amount of seconds.
- Returns
Dict representing DataScanJob. When the job completes with a successful status, information about the Data Profile result is available.
- class airflow.providers.google.cloud.operators.dataplex.DataplexCreateZoneOperator(project_id, region, lake_id, body, zone_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a Zone resource within a Lake.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
body (dict[str, Any] | google.cloud.dataplex_v1.types.Zone) – Required. The Request body contains an instance of Zone.
zone_id (str) – Required. Task identifier.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
Zone
- class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteZoneOperator(project_id, region, lake_id, zone_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Deletes a Zone resource. All assets within a zone must be deleted before the zone can be deleted.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
zone_id (str) – Required. Zone identifier.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
None
- class airflow.providers.google.cloud.operators.dataplex.DataplexCreateAssetOperator(project_id, region, lake_id, body, zone_id, asset_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates an Asset resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the lake belongs to.
zone_id (str) – Required. Zone identifier.
asset_id (str) – Required. Asset identifier.
body (dict[str, Any] | google.cloud.dataplex_v1.types.Asset) – Required. The Request body contains an instance of Asset.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
Asset
- class airflow.providers.google.cloud.operators.dataplex.DataplexDeleteAssetOperator(project_id, region, lake_id, zone_id, asset_id, api_version='v1', retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Deletes an asset resource.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the asset belongs to.
zone_id (str) – Required. Zone identifier.
asset_id (str) – Required. Asset identifier.
api_version (str) – The version of the api that will be requested for example ‘v3’.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- Returns
None