airflow.providers.google.cloud.operators.datastore

This module contains Google Datastore operators.

Module Contents

Classes

CloudDatastoreExportEntitiesOperator

Export entities from Google Cloud Datastore to Cloud Storage

CloudDatastoreImportEntitiesOperator

Import entities from Cloud Storage to Google Cloud Datastore

CloudDatastoreAllocateIdsOperator

Allocate IDs for incomplete keys. Return list of keys.

CloudDatastoreBeginTransactionOperator

Begins a new transaction. Returns a transaction handle.

CloudDatastoreCommitOperator

Commit a transaction, optionally creating, deleting or modifying some entities.

CloudDatastoreRollbackOperator

Roll back a transaction.

CloudDatastoreRunQueryOperator

Run a query for entities. Returns the batch of query results.

CloudDatastoreGetOperationOperator

Gets the latest state of a long-running operation.

CloudDatastoreDeleteOperationOperator

Deletes the long-running operation.
class airflow.providers.google.cloud.operators.datastore.CloudDatastoreExportEntitiesOperator(*, bucket, namespace=None, datastore_conn_id='google_cloud_default', cloud_storage_conn_id='google_cloud_default', delegate_to=None, entity_filter=None, labels=None, polling_interval_in_seconds=10, overwrite_existing=False, project_id=None, impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Export entities from Google Cloud Datastore to Cloud Storage

See also

For more information on how to use this operator, take a look at the guide: Export Entities

See also

https://cloud.google.com/datastore/docs/export-import-entities

Parameters

  • bucket (str) – name of the cloud storage bucket to backup data

  • namespace (Optional[str]) – optional namespace path in the specified Cloud Storage bucket to backup data. If this namespace does not exist in GCS, it will be created.

  • datastore_conn_id (str) – the name of the Datastore connection id to use

  • cloud_storage_conn_id (str) – the name of the cloud storage connection id to force-write backup

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • entity_filter (Optional[dict]) – description of what data from the project is included in the export, refer to https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter

  • labels (Optional[dict]) – client-assigned labels for cloud storage

  • polling_interval_in_seconds (int) – number of seconds to wait before polling for execution status again

  • overwrite_existing (bool) – if the storage bucket + namespace is not empty, it will be emptied prior to exports. This enables overwriting existing backups.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['bucket', 'namespace', 'entity_filter', 'labels', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreImportEntitiesOperator(*, bucket, file, namespace=None, entity_filter=None, labels=None, datastore_conn_id='google_cloud_default', delegate_to=None, polling_interval_in_seconds=10, project_id=None, impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Import entities from Cloud Storage to Google Cloud Datastore

See also

For more information on how to use this operator, take a look at the guide: Import Entities

See also

https://cloud.google.com/datastore/docs/export-import-entities

Parameters

  • bucket (str) – container in Cloud Storage to store data

  • file (str) – path of the backup metadata file in the specified Cloud Storage bucket. It should have the extension .overall_export_metadata

  • namespace (Optional[str]) – optional namespace of the backup metadata file in the specified Cloud Storage bucket.

  • entity_filter (Optional[dict]) – description of what data from the project is included in the export, refer to https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter

  • labels (Optional[dict]) – client-assigned labels for cloud storage

  • datastore_conn_id (str) – the name of the connection id to use

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • polling_interval_in_seconds (float) – number of seconds to wait before polling for execution status again

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['bucket', 'file', 'namespace', 'entity_filter', 'labels', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreAllocateIdsOperator(*, partial_keys, project_id=None, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Allocate IDs for incomplete keys. Return list of keys.

See also

For more information on how to use this operator, take a look at the guide: Allocate Ids

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds

Parameters

  • partial_keys (List) – a list of partial keys.

  • project_id (Optional[str]) – Google Cloud project ID against which to make the request.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['partial_keys', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreBeginTransactionOperator(*, transaction_options, project_id=None, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Begins a new transaction. Returns a transaction handle.

See also

For more information on how to use this operator, take a look at the guide: Begin transaction

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction

Parameters

  • transaction_options (Dict[str, Any]) – Options for a new transaction.

  • project_id (Optional[str]) – Google Cloud project ID against which to make the request.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['transaction_options', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreCommitOperator(*, body, project_id=None, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Commit a transaction, optionally creating, deleting or modifying some entities.

See also

For more information on how to use this operator, take a look at the guide: Commit transaction

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit

Parameters

  • body (Dict[str, Any]) – the body of the commit request.

  • project_id (Optional[str]) – Google Cloud project ID against which to make the request.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['body', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreRollbackOperator(*, transaction, project_id=None, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Roll back a transaction.

See also

For more information on how to use this operator, take a look at the guide: Roll back transaction

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback

Parameters

  • transaction (str) – the transaction to roll back.

  • project_id (Optional[str]) – Google Cloud project ID against which to make the request.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['transaction', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreRunQueryOperator(*, body, project_id=None, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Run a query for entities. Returns the batch of query results.

See also

For more information on how to use this operator, take a look at the guide: Run query

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery

Parameters

  • body (Dict[str, Any]) – the body of the query request.

  • project_id (Optional[str]) – Google Cloud project ID against which to make the request.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['body', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreGetOperationOperator(*, name, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Gets the latest state of a long-running operation.

See also

For more information on how to use this operator, take a look at the guide: Get operation state

See also

https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/get

Parameters

  • name (str) – the name of the operation resource.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['name', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreDeleteOperationOperator(*, name, delegate_to=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Deletes the long-running operation.

See also

For more information on how to use this operator, take a look at the guide: Delete operation

See also

https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/delete

Parameters

  • name (str) – the name of the operation resource.

  • delegate_to (Optional[str]) – The account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (Optional[Union[str, Sequence[str]]]) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields :Sequence[str] = ['name', 'impersonation_chain'][source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?