airflow.contrib.hooks.gcp_transfer_hook¶
Module Contents¶
-
airflow.contrib.hooks.gcp_transfer_hook.ALREADY_EXISTING_IN_SINK= overwriteObjectsAlreadyExistingInSink[source]¶
-
airflow.contrib.hooks.gcp_transfer_hook.TRANSFER_JOB_FIELD_MASK= update_transfer_job_field_mask[source]¶
-
class
airflow.contrib.hooks.gcp_transfer_hook.GCPTransferServiceHook(api_version='v1', gcp_conn_id='google_cloud_default', delegate_to=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHookHook for Google Storage Transfer Service.
-
get_conn(self)[source]¶ Retrieves connection to Google Storage Transfer service.
- Returns
Google Storage Transfer service object
- Return type
-
create_transfer_job(self, body)[source]¶ Creates a transfer job that runs periodically.
- Parameters
body (dict) – (Required) A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
- Returns
transfer job. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
- Return type
-
get_transfer_job(self, job_name, project_id=None)[source]¶ Gets the latest state of a long-running operation in Google Storage Transfer Service.
-
list_transfer_job(self, filter)[source]¶ Lists long-running operations in Google Storage Transfer Service that match the specified filter.
- Parameters
filter (dict) – (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
- Returns
List of Transfer Jobs
- Return type
-
update_transfer_job(self, job_name, body)[source]¶ Updates a transfer job that runs periodically.
- Parameters
job_name (str) – (Required) Name of the job to be updated
body (dict) – A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
- Returns
If successful, TransferJob.
- Return type
-
delete_transfer_job(self, job_name, project_id)[source]¶ Deletes a transfer job. This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collection. Transfer jobs become eligible for garbage collection 30 days after soft delete.
-
cancel_transfer_operation(self, operation_name)[source]¶ Cancels an transfer operation in Google Storage Transfer Service.
-
get_transfer_operation(self, operation_name)[source]¶ Gets an transfer operation in Google Storage Transfer Service.
- Parameters
operation_name (str) – (Required) Name of the transfer operation.
- Returns
transfer operation See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation
- Return type
-
list_transfer_operations(self, filter)[source]¶ Gets an transfer operation in Google Storage Transfer Service.
- Parameters
filter (dict) –
(Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter With one additional improvement:
project_id is optional if you have a project id defined in the connection See: Google Cloud Platform Connection
- Returns
transfer operation
- Return type
-
pause_transfer_operation(self, operation_name)[source]¶ Pauses an transfer operation in Google Storage Transfer Service.
-
resume_transfer_operation(self, operation_name)[source]¶ Resumes an transfer operation in Google Storage Transfer Service.
-
wait_for_transfer_job(self, job, expected_statuses=(GcpTransferOperationStatus.SUCCESS, ), timeout=60)[source]¶ Waits until the job reaches the expected state.
- Parameters
job (dict) – Transfer job See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
expected_statuses (set[str]) – State that is expected See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
timeout (time in which the operation must end in seconds) –
- Return type
-
static
operations_contain_expected_statuses(operations, expected_statuses)[source]¶ Checks whether the operation list has an operation with the expected status, then returns true If it encounters operations in FAILED or ABORTED state throw
airflow.exceptions.AirflowException.- Parameters
operations (list[dict]) – (Required) List of transfer operations to check.
expected_statuses (set[str]) – (Required) status that is expected See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
- Returns
If there is an operation with the expected state in the operation list, returns true,
- Raises
airflow.exceptions.AirflowException If it encounters operations with a state in the list,
- Return type
-