airflow.providers.google.cloud.hooks.dataproc_metastore

This module contains a Google Cloud Dataproc Metastore hook.

Module Contents

class airflow.providers.google.cloud.hooks.dataproc_metastore.DataprocMetastoreHook[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud Dataproc Metastore APIs.

get_dataproc_metastore_client(self)[source]

Returns DataprocMetastoreClient.

wait_for_operation(self, timeout: float, operation: Operation)[source]

Waits for long-lasting operation to complete.

create_backup(self, project_id: str, region: str, service_id: str, backup: Backup, backup_id: str, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Creates a new backup in a given project and location.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • backup (google.cloud.metastore_v1.types.Backup) --

    Required. The backup to create. The name field is ignored. The ID of the created backup must be provided in the request's backup_id field.

    This corresponds to the backup field on the request instance; if request is provided, this should not be set.

  • backup_id (str) --

    Required. The ID of the backup, which is used as the final component of the backup's name. This value must be between 1 and 64 characters long, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the backup_id field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

create_metadata_import(self, project_id: str, region: str, service_id: str, metadata_import: MetadataImport, metadata_import_id: str, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Creates a new MetadataImport in a given project and location.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • metadata_import (google.cloud.metastore_v1.types.MetadataImport) --

    Required. The metadata import to create. The name field is ignored. The ID of the created metadata import must be provided in the request's metadata_import_id field.

    This corresponds to the metadata_import field on the request instance; if request is provided, this should not be set.

  • metadata_import_id (str) --

    Required. The ID of the metadata import, which is used as the final component of the metadata import's name. This value must be between 1 and 64 characters long, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the metadata_import_id field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

create_service(self, region: str, project_id: str, service: Union[Dict, Service], service_id: str, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = ())[source]

Creates a metastore service in a project and location.

Parameters

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • service (google.cloud.metastore_v1.types.Service) --

    Required. The Metastore service to create. The name field is ignored. The ID of the created metastore service must be provided in the request's service_id field.

    This corresponds to the service field on the request instance; if request is provided, this should not be set.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

delete_backup(self, project_id: str, region: str, service_id: str, backup_id: str, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Deletes a single backup.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • backup_id (str) --

    Required. The ID of the backup, which is used as the final component of the backup's name. This value must be between 1 and 64 characters long, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the backup_id field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

delete_service(self, project_id: str, region: str, service_id: str, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Deletes a single service.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

export_metadata(self, destination_gcs_folder: str, project_id: str, region: str, service_id: str, request_id: Optional[str] = None, database_dump_type: Optional[DatabaseDumpSpec] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Exports metadata from a service.

Parameters

  • destination_gcs_folder (str) -- A Cloud Storage URI of a folder, in the format gs://<bucket_name>/<path_inside_bucket>. A sub-folder <export_folder> containing exported files will be created below it.

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • database_dump_type (google.cloud.metastore_v1.types.DatabaseDumpSpec.Type) -- Optional. The type of the database dump. If unspecified, defaults to MYSQL.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

get_service(self, project_id: str, region: str, service_id: str, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Gets the details of a single service.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

list_backups(self, project_id: str, region: str, service_id: str, page_size: Optional[int] = None, page_token: Optional[str] = None, filter: Optional[str] = None, order_by: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Lists backups in a service.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • page_size (int) -- Optional. The maximum number of backups to return. The response may contain less than the maximum number. If unspecified, no more than 500 backups are returned. The maximum value is 1000; values above 1000 are changed to 1000.

  • page_token (str) -- Optional. A page token, received from a previous [DataprocMetastore.ListBackups][google.cloud.metastore.v1.DataprocMetastore.ListBackups] call. Provide this token to retrieve the subsequent page. To retrieve the first page, supply an empty page token. When paginating, other parameters provided to [DataprocMetastore.ListBackups][google.cloud.metastore.v1.DataprocMetastore.ListBackups] must match the call that provided the page token.

  • filter (str) -- Optional. The filter to apply to list results.

  • order_by (str) -- Optional. Specify the ordering of results as described in Sorting Order. If not specified, the results will be sorted in the default order.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

restore_service(self, project_id: str, region: str, service_id: str, backup_project_id: str, backup_region: str, backup_service_id: str, backup_id: str, restore_type: Optional[Restore] = None, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Restores a service from a backup.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • backup_project_id (str) -- Required. The ID of the Google Cloud project that the metastore service backup to restore from.

  • backup_region (str) -- Required. The ID of the Google Cloud region that the metastore service backup to restore from.

  • backup_service_id (str) -- Required. The ID of the metastore service backup to restore from, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

  • backup_id (str) -- Required. The ID of the metastore service backup to restore from

  • restore_type (google.cloud.metastore_v1.types.Restore.RestoreType) -- Optional. The type of restore. If unspecified, defaults to METADATA_ONLY

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

update_service(self, project_id: str, region: str, service_id: str, service: Union[Dict, Service], update_mask: FieldMask, request_id: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]

Updates the parameters of a single service.

Parameters

  • project_id (str) -- Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) -- Required. The ID of the Google Cloud region that the service belongs to.

  • service_id (str) --

    Required. The ID of the metastore service, which is used as the final component of the metastore service's name. This value must be between 2 and 63 characters long inclusive, begin with a letter, end with a letter or number, and consist of alphanumeric ASCII characters or hyphens.

    This corresponds to the service_id field on the request instance; if request is provided, this should not be set.

  • service (Union[Dict, google.cloud.metastore_v1.types.Service]) --

    Required. The metastore service to update. The server only merges fields in the service if they are specified in update_mask.

    The metastore service's name field is used to identify the metastore service to be updated.

    This corresponds to the service field on the request instance; if request is provided, this should not be set.

  • update_mask (google.protobuf.field_mask_pb2.FieldMask) --

    Required. A field mask used to specify the fields to be overwritten in the metastore service resource by the update. Fields specified in the update_mask are relative to the resource (not to the full request). A field is overwritten if it is in the mask.

    This corresponds to the update_mask field on the request instance; if request is provided, this should not be set.

  • request_id (str) -- Optional. A unique id used to identify the request.

  • retry (google.api_core.retry.Retry) -- Designation of what errors, if any, should be retried.

  • timeout (float) -- The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) -- Strings which should be sent along with the request as metadata.

Was this entry helpful?