airflow.providers.google.cloud.hooks.cloud_storage_transfer_service
¶
This module contains a Google Storage Transfer Service Hook.
Module Contents¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TIME_TO_SLEEP_IN_SECONDS
= 10[source]¶
-
class
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
GcpTransferJobsStatus
[source]¶ Class with Google Cloud Transfer jobs statuses.
-
class
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
GcpTransferOperationStatus
[source]¶ Class with Google Cloud Transfer operations statuses.
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
ACCESS_KEY_ID
= accessKeyId[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
ALREADY_EXISTING_IN_SINK
= overwriteObjectsAlreadyExistingInSink[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
AWS_ACCESS_KEY
= awsAccessKey[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
AWS_S3_DATA_SOURCE
= awsS3DataSource[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
BUCKET_NAME
= bucketName[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
DESCRIPTION
= description[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
FILTER_JOB_NAMES
= job_names[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
FILTER_PROJECT_ID
= project_id[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
GCS_DATA_SINK
= gcsDataSink[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
GCS_DATA_SOURCE
= gcsDataSource[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
HTTP_DATA_SOURCE
= httpDataSource[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
OBJECT_CONDITIONS
= object_conditions[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
OPERATIONS
= operations[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
SCHEDULE_END_DATE
= scheduleEndDate[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
SCHEDULE_START_DATE
= scheduleStartDate[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
SECRET_ACCESS_KEY
= secretAccessKey[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
START_TIME_OF_DAY
= startTimeOfDay[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TRANSFER_JOB
= transfer_job[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TRANSFER_JOBS
= transferJobs[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TRANSFER_JOB_FIELD_MASK
= update_transfer_job_field_mask[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TRANSFER_OPERATIONS
= transferOperations[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TRANSFER_OPTIONS
= transfer_options[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
TRANSFER_SPEC
= transferSpec[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
ALREADY_EXIST_CODE
= 409[source]¶
-
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
gen_job_name
(job_name: str) → str[source]¶ -
Adds unique suffix to job name. If suffix already exists, updates it.
-
Suffix — current timestamp
- Parameters
job_name --
- Rtype job_name
str
- Returns
job_name with suffix
- Return type
-
class
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.
CloudDataTransferServiceHook
(api_version: str = 'v1', gcp_conn_id: str = 'google_cloud_default', delegate_to: Optional[str] = None, impersonation_chain: Optional[Union[str, Sequence[str]]] = None)[source]¶ Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseHook
Hook for Google Storage Transfer Service.
All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.
-
get_conn
(self)[source]¶ Retrieves connection to Google Storage Transfer service.
- Returns
Google Storage Transfer service object
- Return type
-
create_transfer_job
(self, body: dict)[source]¶ Creates a transfer job that runs periodically.
- Parameters
body (dict) -- (Required) A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
- Returns
transfer job. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
- Return type
-
get_transfer_job
(self, job_name: str, project_id: str)[source]¶ Gets the latest state of a long-running operation in Google Storage Transfer Service.
-
list_transfer_job
(self, request_filter: Optional[dict] = None, **kwargs)[source]¶ Lists long-running operations in Google Storage Transfer Service that match the specified filter.
- Parameters
request_filter (dict) -- (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
- Returns
List of Transfer Jobs
- Return type
-
enable_transfer_job
(self, job_name: str, project_id: str)[source]¶ New transfers will be performed based on the schedule.
-
update_transfer_job
(self, job_name: str, body: dict)[source]¶ Updates a transfer job that runs periodically.
- Parameters
job_name (str) -- (Required) Name of the job to be updated
body (dict) -- A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
- Returns
If successful, TransferJob.
- Return type
-
delete_transfer_job
(self, job_name: str, project_id: str)[source]¶ Deletes a transfer job. This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collection. Transfer jobs become eligible for garbage collection 30 days after soft delete.
-
cancel_transfer_operation
(self, operation_name: str)[source]¶ Cancels an transfer operation in Google Storage Transfer Service.
-
get_transfer_operation
(self, operation_name: str)[source]¶ Gets an transfer operation in Google Storage Transfer Service.
- Parameters
operation_name (str) -- (Required) Name of the transfer operation.
- Returns
transfer operation See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation
- Return type
-
list_transfer_operations
(self, request_filter: Optional[dict] = None, **kwargs)[source]¶ Gets an transfer operation in Google Storage Transfer Service.
- Parameters
request_filter (dict) --
(Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter With one additional improvement:
project_id is optional if you have a project id defined in the connection See: Google Cloud Connection
- Returns
transfer operation
- Return type
-
pause_transfer_operation
(self, operation_name: str)[source]¶ Pauses an transfer operation in Google Storage Transfer Service.
-
resume_transfer_operation
(self, operation_name: str)[source]¶ Resumes an transfer operation in Google Storage Transfer Service.
-
wait_for_transfer_job
(self, job: dict, expected_statuses: Optional[Set[str]] = None, timeout: Optional[Union[float, timedelta]] = None)[source]¶ Waits until the job reaches the expected state.
- Parameters
job (dict) -- Transfer job See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
expected_statuses (set[str]) -- State that is expected See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
timeout (Optional[Union[float, timedelta]]) -- Time in which the operation must end in seconds. If not specified, defaults to 60 seconds.
- Return type
-
static
operations_contain_expected_statuses
(operations: List[dict], expected_statuses: Union[Set[str], str])[source]¶ Checks whether the operation list has an operation with the expected status, then returns true If it encounters operations in FAILED or ABORTED state throw
airflow.exceptions.AirflowException
.- Parameters
operations (list[dict]) -- (Required) List of transfer operations to check.
expected_statuses (set[str]) -- (Required) status that is expected See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
- Returns
If there is an operation with the expected state in the operation list, returns true,
- Raises
airflow.exceptions.AirflowException If it encounters operations with a state in the list,
- Return type
-