airflow.providers.google.cloud.hooks.bigquery_dts
¶
This module contains a BigQuery Hook.
Module Contents¶
-
airflow.providers.google.cloud.hooks.bigquery_dts.
get_object_id
(obj: dict) → str[source]¶ -
Returns unique id of the object.
-
class
airflow.providers.google.cloud.hooks.bigquery_dts.
BiqQueryDataTransferServiceHook
(gcp_conn_id: str = 'google_cloud_default', delegate_to: Optional[str] = None, impersonation_chain: Optional[Union[str, Sequence[str]]] = None)[source]¶ Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseHook
Hook for Google Bigquery Transfer API.
All the methods in the hook where
project_id
is used must be called with keyword arguments rather than positional.-
static
_disable_auto_scheduling
(config: Union[dict, TransferConfig])[source]¶ In the case of Airflow, the customer needs to create a transfer config with the automatic scheduling disabled (UI, CLI or an Airflow operator) and then trigger a transfer run using a specialized Airflow operator that will call start_manual_transfer_runs.
- Parameters
config (Union[dict, google.cloud.bigquery_datatransfer_v1.types.TransferConfig]) – Data transfer configuration to create.
-
get_conn
(self)[source]¶ Retrieves connection to Google Bigquery.
- Returns
Google Bigquery API client
- Return type
google.cloud.bigquery_datatransfer_v1.DataTransferServiceClient
-
create_transfer_config
(self, transfer_config: Union[dict, TransferConfig], project_id: str, authorization_code: Optional[str] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]¶ Creates a new data transfer configuration.
- Parameters
transfer_config (Union[dict, google.cloud.bigquery_datatransfer_v1.types.TransferConfig]) – Data transfer configuration to create.
project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.
authorization_code (Optional[str]) – authorization code to use with this transfer configuration. This is required if new credentials are needed.
retry (Optional[google.api_core.retry.Retry]) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (Optional[float]) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Optional[Sequence[Tuple[str, str]]]) – Additional metadata that is provided to the method.
- Returns
A
google.cloud.bigquery_datatransfer_v1.types.TransferConfig
instance.
-
delete_transfer_config
(self, transfer_config_id: str, project_id: str, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]¶ Deletes transfer configuration.
- Parameters
transfer_config_id (str) – Id of transfer config to be used.
project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.
retry (Optional[google.api_core.retry.Retry]) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (Optional[float]) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Optional[Sequence[Tuple[str, str]]]) – Additional metadata that is provided to the method.
- Returns
None
-
start_manual_transfer_runs
(self, transfer_config_id: str, project_id: str, requested_time_range: Optional[dict] = None, requested_run_time: Optional[dict] = None, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]¶ Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.
- Parameters
transfer_config_id (str) – Id of transfer config to be used.
requested_time_range (Union[dict, TimeRange]) – Time range for the transfer runs that should be started. If a dict is provided, it must be of the same form as the protobuf message ~google.cloud.bigquery_datatransfer_v1.types.TimeRange
requested_run_time (Union[dict, Timestamp]) – Specific run_time for a transfer run to be started. The requested_run_time must not be in the future. If a dict is provided, it must be of the same form as the protobuf message ~google.cloud.bigquery_datatransfer_v1.types.Timestamp
project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.
retry (Optional[google.api_core.retry.Retry]) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (Optional[float]) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Optional[Sequence[Tuple[str, str]]]) – Additional metadata that is provided to the method.
- Returns
An
google.cloud.bigquery_datatransfer_v1.types.StartManualTransferRunsResponse
instance.
-
get_transfer_run
(self, run_id: str, transfer_config_id: str, project_id: str, retry: Optional[Retry] = None, timeout: Optional[float] = None, metadata: Optional[Sequence[Tuple[str, str]]] = None)[source]¶ Returns information about the particular transfer run.
- Parameters
run_id (str) – ID of the transfer run.
transfer_config_id (str) – ID of transfer config to be used.
project_id (str) – The BigQuery project id where the transfer configuration should be created. If set to None or missing, the default project_id from the Google Cloud connection is used.
retry (Optional[google.api_core.retry.Retry]) – A retry object used to retry requests. If None is specified, requests will not be retried.
timeout (Optional[float]) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.
metadata (Optional[Sequence[Tuple[str, str]]]) – Additional metadata that is provided to the method.
- Returns
An
google.cloud.bigquery_datatransfer_v1.types.TransferRun
instance.
-
static