airflow.providers.google.cloud.hooks.datastore

This module contains Google Datastore hook.

Module Contents

class airflow.providers.google.cloud.hooks.datastore.DatastoreHook(gcp_conn_id: str = 'google_cloud_default', delegate_to: Optional[str] = None, api_version: str = 'v1', datastore_conn_id: Optional[str] = None, impersonation_chain: Optional[Union[str, Sequence[str]]] = None)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Interact with Google Cloud Datastore. This hook uses the Google Cloud connection.

This object is not threads safe. If you want to make multiple requests simultaneously, you will need to create a hook per thread.

Parameters

api_version (str) -- The version of the API it is going to connect to.

get_conn(self)[source]

Establishes a connection to the Google API.

Returns

a Google Cloud Datastore service object.

Return type

Resource

allocate_ids(self, partial_keys: list, project_id: str)[source]

Allocate IDs for incomplete keys.

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds

Parameters

  • partial_keys (list) -- a list of partial keys.

  • project_id (str) -- Google Cloud project ID against which to make the request.

Returns

a list of full keys.

Return type

list

begin_transaction(self, project_id: str, transaction_options: Dict[str, Any])[source]

Begins a new transaction.

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction

Parameters

  • project_id (str) -- Google Cloud project ID against which to make the request.

  • transaction_options (Dict[str, Any]) -- Options for a new transaction.

Returns

a transaction handle.

Return type

str

commit(self, body: dict, project_id: str)[source]

Commit a transaction, optionally creating, deleting or modifying some entities.

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit

Parameters

  • body (dict) -- the body of the commit request.

  • project_id (str) -- Google Cloud project ID against which to make the request.

Returns

the response body of the commit request.

Return type

dict

lookup(self, keys: list, project_id: str, read_consistency: Optional[str] = None, transaction: Optional[str] = None)[source]

Lookup some entities by key.

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/lookup

Parameters

  • keys (list) -- the keys to lookup.

  • read_consistency (str) -- the read consistency to use. default, strong or eventual. Cannot be used with a transaction.

  • transaction (str) -- the transaction to use, if any.

  • project_id (str) -- Google Cloud project ID against which to make the request.

Returns

the response body of the lookup request.

Return type

dict

rollback(self, transaction: str, project_id: str)[source]

Roll back a transaction.

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback

Parameters

  • transaction (str) -- the transaction to roll back.

  • project_id (str) -- Google Cloud project ID against which to make the request.

run_query(self, body: dict, project_id: str)[source]

Run a query for entities.

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery

Parameters

  • body (dict) -- the body of the query request.

  • project_id (str) -- Google Cloud project ID against which to make the request.

Returns

the batch of query results.

Return type

dict

get_operation(self, name: str)[source]

Gets the latest state of a long-running operation.

See also

https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/get

Parameters

name (str) -- the name of the operation resource.

Returns

a resource operation instance.

Return type

dict

delete_operation(self, name: str)[source]

Deletes the long-running operation.

See also

https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/delete

Parameters

name (str) -- the name of the operation resource.

Returns

none if successful.

Return type

dict

poll_operation_until_done(self, name: str, polling_interval_in_seconds: int)[source]

Poll backup operation state until it's completed.

Parameters

  • name (str) -- the name of the operation resource

  • polling_interval_in_seconds (int) -- The number of seconds to wait before calling another request.

Returns

a resource operation instance.

Return type

dict

export_to_storage_bucket(self, bucket: str, project_id: str, namespace: Optional[str] = None, entity_filter: Optional[dict] = None, labels: Optional[Dict[str, str]] = None)[source]

Export entities from Cloud Datastore to Cloud Storage for backup.

Note

Keep in mind that this requests the Admin API not the Data API.

See also

https://cloud.google.com/datastore/docs/reference/admin/rest/v1/projects/export

Parameters

  • bucket (str) -- The name of the Cloud Storage bucket.

  • namespace (str) -- The Cloud Storage namespace path.

  • entity_filter (dict) -- Description of what data from the project is included in the export.

  • labels (dict of str) -- Client-assigned labels.

  • project_id (str) -- Google Cloud project ID against which to make the request.

Returns

a resource operation instance.

Return type

dict

import_from_storage_bucket(self, bucket: str, file: str, project_id: str, namespace: Optional[str] = None, entity_filter: Optional[dict] = None, labels: Optional[Union[dict, str]] = None)[source]

Import a backup from Cloud Storage to Cloud Datastore.

Note

Keep in mind that this requests the Admin API not the Data API.

See also

https://cloud.google.com/datastore/docs/reference/admin/rest/v1/projects/import

Parameters

  • bucket (str) -- The name of the Cloud Storage bucket.

  • file (str) -- the metadata file written by the projects.export operation.

  • namespace (str) -- The Cloud Storage namespace path.

  • entity_filter (dict) -- specify which kinds/namespaces are to be imported.

  • labels (dict of str) -- Client-assigned labels.

  • project_id (str) -- Google Cloud project ID against which to make the request.

Returns

a resource operation instance.

Return type

dict

Was this entry helpful?