`airflow.providers.google.cloud.hooks.dataplex`¶

This module contains Google Dataplex hook.

Module Contents¶

Classes¶

`DataplexHook`	Hook for Google Dataplex.
`DataplexAsyncHook`	Asynchronous Hook for Google Cloud Dataplex APIs.

Attributes¶

PATH_DATA_SCAN

airflow.providers.google.cloud.hooks.dataplex.PATH_DATA_SCAN = 'projects/{project_id}/locations/{region}/dataScans/{data_scan_id}'[source]¶

exception airflow.providers.google.cloud.hooks.dataplex.AirflowDataQualityScanException[source]¶

Bases: airflow.exceptions.AirflowException

Raised when data quality scan rules fail.

exception airflow.providers.google.cloud.hooks.dataplex.AirflowDataQualityScanResultTimeoutException[source]¶

Bases: airflow.exceptions.AirflowException

Raised when no result found after specified amount of seconds.

class airflow.providers.google.cloud.hooks.dataplex.DataplexHook(api_version='v1', gcp_conn_id='google_cloud_default', impersonation_chain=None, location=None, **kwargs)[source]¶

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Dataplex.

Parameters

api_version (str) – The version of the api that will be requested for example ‘v3’.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

get_dataplex_client()[source]¶

Returns DataplexServiceClient.

get_dataplex_data_scan_client()[source]¶

Returns DataScanServiceClient.

wait_for_operation(timeout, operation)[source]¶

Waits for long-lasting operation to complete.

create_task(project_id, region, lake_id, body, dataplex_task_id, validate_only=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Creates a task resource within a lake.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
body (dict[str, Any] | google.cloud.dataplex_v1.types.Task) – Required. The Request body contains an instance of Task.
dataplex_task_id (str) – Required. Task identifier.
validate_only (bool | None) – Optional. Only validate the request, but do not perform mutations. The default is false.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

delete_task(project_id, region, lake_id, dataplex_task_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Delete the task resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
dataplex_task_id (str) – Required. The ID of the Google Cloud task to be deleted.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

list_tasks(project_id, region, lake_id, page_size=None, page_token=None, filter=None, order_by=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Lists tasks under the given lake.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
page_size (int | None) – Optional. Maximum number of tasks to return. The service may return fewer than this value. If unspecified, at most 10 tasks will be returned. The maximum value is 1000; values above 1000 will be coerced to 1000.
page_token (str | None) – Optional. Page token received from a previous ListZones call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListZones must match the call that provided the page token.
filter (str | None) – Optional. Filter request.
order_by (str | None) – Optional. Order by fields for the result.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

get_task(project_id, region, lake_id, dataplex_task_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Get task resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake that the task belongs to.
dataplex_task_id (str) – Required. The ID of the Google Cloud task to be retrieved.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

delete_lake(project_id, region, lake_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Delete the lake resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake to be deleted.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

create_lake(project_id, region, lake_id, body, validate_only=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Creates a lake resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. Lake identifier.
body (dict[str, Any] | google.cloud.dataplex_v1.types.Lake) – Required. The Request body contains an instance of Lake.
validate_only (bool | None) – Optional. Only validate the request, but do not perform mutations. The default is false.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

get_lake(project_id, region, lake_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Get lake resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake to be retrieved.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

create_zone(project_id, region, lake_id, zone_id, body, retry=DEFAULT, timeout=None, metadata=())[source]¶

Creates a zone resource within a lake.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake to be retrieved.
body (dict[str, Any] | google.cloud.dataplex_v1.types.Zone) – Required. The Request body contains an instance of Zone.
zone_id (str) – Required. Zone identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

delete_zone(project_id, region, lake_id, zone_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Deletes a zone resource. All assets within a zone must be deleted before the zone can be deleted.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake to be retrieved.
zone_id (str) – Required. Zone identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

create_asset(project_id, region, lake_id, zone_id, asset_id, body, retry=DEFAULT, timeout=None, metadata=())[source]¶

Creates an asset resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake to be retrieved.
zone_id (str) – Required. Zone identifier.
asset_id (str) – Required. Asset identifier.
body (dict[str, Any] | google.cloud.dataplex_v1.types.Asset) – Required. The Request body contains an instance of Asset.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

delete_asset(project_id, region, lake_id, asset_id, zone_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Deletes an asset resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
lake_id (str) – Required. The ID of the Google Cloud lake to be retrieved.
zone_id (str) – Required. Zone identifier.
asset_id (str) – Required. Asset identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

create_data_scan(project_id, region, body, data_scan_id=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Creates a DataScan resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str | None) – Required. Data Quality scan identifier.
body (dict[str, Any] | google.cloud.dataplex_v1.types.DataScan) – Required. The Request body contains an instance of DataScan.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

run_data_scan(project_id, region, data_scan_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Runs an on-demand execution of a DataScan.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

get_data_scan_job(project_id, region, data_scan_id=None, job_id=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Gets a DataScan Job resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str | None) – Required. Data Quality scan identifier.
job_id (str | None) – Required. The resource name of the DataScanJob: projects/{project_id}/locations/{region}/dataScans/{data_scan_id}/jobs/{data_scan_job_id}
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

wait_for_data_scan_job(data_scan_id, job_id=None, project_id=None, region=None, wait_time=10, result_timeout=None)[source]¶

Wait for Dataplex data scan job.

Parameters

job_id (str | None) – Required. The job_id to wait for.
data_scan_id (str) – Required. Data Quality scan identifier.
region (str | None) – Required. The ID of the Google Cloud region that the lake belongs to.
project_id (str | None) – Optional. Google Cloud project ID.
wait_time (int) – Number of seconds between checks.
result_timeout (float | None) – Value in seconds for which operator will wait for the Data Quality scan result. Throws exception if there is no result found after specified amount of seconds.

get_data_scan(project_id, region, data_scan_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Gets a DataScan resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

update_data_scan(project_id, region, data_scan_id, body, update_mask=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Updates a DataScan resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
body (dict[str, Any] | google.cloud.dataplex_v1.types.DataScan) – Required. The Request body contains an instance of DataScan.
update_mask (dict | google.protobuf.field_mask_pb2.FieldMask | None) – Required. Mask of fields to update.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

delete_data_scan(project_id, region, data_scan_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Deletes a DataScan resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

list_data_scan_jobs(project_id, region, data_scan_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Lists DataScanJobs under the given DataScan.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str) – Required. Data Quality scan identifier.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

class airflow.providers.google.cloud.hooks.dataplex.DataplexAsyncHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseAsyncHook

Asynchronous Hook for Google Cloud Dataplex APIs.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

sync_hook_class[source]¶

async get_dataplex_data_scan_client()[source]¶

Returns DataScanServiceAsyncClient.

async get_data_scan_job(project_id, region, data_scan_id=None, job_id=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Gets a DataScan Job resource.

Parameters

project_id (str) – Required. The ID of the Google Cloud project that the lake belongs to.
region (str) – Required. The ID of the Google Cloud region that the lake belongs to.
data_scan_id (str | None) – Required. DataScan identifier.
job_id (str | None) – Required. The resource name of the DataScanJob: projects/{project_id}/locations/{region}/dataScans/{data_scan_id}/jobs/{data_scan_job_id}
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

airflow.providers.google.cloud.hooks.dataplex¶

Module Contents¶

Classes¶

Attributes¶

`airflow.providers.google.cloud.hooks.dataplex`¶