airflow.providers.google.cloud.hooks.bigquery_dts

This module contains a BigQuery Hook.

Module Contents

Classes

BiqQueryDataTransferServiceHook

Hook for Google Bigquery Transfer API.

AsyncBiqQueryDataTransferServiceHook

Hook of the BigQuery service to be used with async client of the Google library.

Functions

get_object_id(obj)

Return unique id of the object.
airflow.providers.google.cloud.hooks.bigquery_dts.get_object_id(obj)[source]

Return unique id of the object.

class airflow.providers.google.cloud.hooks.bigquery_dts.BiqQueryDataTransferServiceHook(gcp_conn_id='google_cloud_default', location=None, impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Bigquery Transfer API.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

get_conn()[source]

Retrieve connection to Google Bigquery.

Returns

Google Bigquery API client

Return type

google.cloud.bigquery_datatransfer_v1.DataTransferServiceClient

create_transfer_config(transfer_config, project_id=PROVIDE_PROJECT_ID, authorization_code=None, retry=DEFAULT, timeout=None, metadata=())[source]

Create a new data transfer configuration.

Parameters

  • transfer_config (dict | google.cloud.bigquery_datatransfer_v1.types.TransferConfig) – Data transfer configuration to create.

  • project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • authorization_code (str | None) – authorization code to use with this transfer configuration. This is required if new credentials are needed.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

Returns

A google.cloud.bigquery_datatransfer_v1.types.TransferConfig instance.

Return type

google.cloud.bigquery_datatransfer_v1.types.TransferConfig

delete_transfer_config(transfer_config_id, project_id=PROVIDE_PROJECT_ID, retry=DEFAULT, timeout=None, metadata=())[source]

Delete transfer configuration.

Parameters

  • transfer_config_id (str) – Id of transfer config to be used.

  • project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

Returns

None

Return type

None

start_manual_transfer_runs(transfer_config_id, project_id=PROVIDE_PROJECT_ID, requested_time_range=None, requested_run_time=None, retry=DEFAULT, timeout=None, metadata=())[source]

Start manual transfer runs to be executed now with schedule_time equal to current time.

The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.

Parameters

  • transfer_config_id (str) – Id of transfer config to be used.

  • requested_time_range (dict | None) – Time range for the transfer runs that should be started. If a dict is provided, it must be of the same form as the protobuf message ~google.cloud.bigquery_datatransfer_v1.types.TimeRange

  • requested_run_time (dict | None) – Specific run_time for a transfer run to be started. The requested_run_time must not be in the future. If a dict is provided, it must be of the same form as the protobuf message ~google.cloud.bigquery_datatransfer_v1.types.Timestamp

  • project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

Returns

An google.cloud.bigquery_datatransfer_v1.types.StartManualTransferRunsResponse instance.

Return type

google.cloud.bigquery_datatransfer_v1.types.StartManualTransferRunsResponse

get_transfer_run(run_id, transfer_config_id, project_id=PROVIDE_PROJECT_ID, retry=DEFAULT, timeout=None, metadata=())[source]

Return information about the particular transfer run.

Parameters

  • run_id (str) – ID of the transfer run.

  • transfer_config_id (str) – ID of transfer config to be used.

  • project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

Returns

An google.cloud.bigquery_datatransfer_v1.types.TransferRun instance.

Return type

google.cloud.bigquery_datatransfer_v1.types.TransferRun

class airflow.providers.google.cloud.hooks.bigquery_dts.AsyncBiqQueryDataTransferServiceHook(gcp_conn_id='google_cloud_default', location=None, impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseAsyncHook

Hook of the BigQuery service to be used with async client of the Google library.

sync_hook_class[source]
async get_transfer_run(config_id, run_id, project_id, location=None, retry=DEFAULT, timeout=None, metadata=())[source]

Return information about the particular transfer run.

Parameters

  • run_id (str) – ID of the transfer run.

  • config_id (str) – ID of transfer config to be used.

  • project_id (str | None) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • location (str | None) – BigQuery Transfer Service location for regional transfers.

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.

Returns

An google.cloud.bigquery_datatransfer_v1.types.TransferRun instance.

Was this entry helpful?