airflow.providers.google.cloud.hooks.cloud_storage_transfer_service

This module contains a Google Storage Transfer Service Hook.

Module Contents

Classes

GcpTransferJobsStatus

Class with Google Cloud Transfer jobs statuses.

GcpTransferOperationStatus

Class with Google Cloud Transfer operations statuses.

CloudDataTransferServiceHook

Hook for Google Storage Transfer Service.

Functions

gen_job_name(job_name)

Adds unique suffix to job name. If suffix already exists, updates it.

Attributes

log

TIME_TO_SLEEP_IN_SECONDS

ACCESS_KEY_ID

ALREADY_EXISTING_IN_SINK

AWS_ACCESS_KEY

AWS_S3_DATA_SOURCE

BODY

BUCKET_NAME

COUNTERS

DAY

DESCRIPTION

FILTER

FILTER_JOB_NAMES

FILTER_PROJECT_ID

GCS_DATA_SINK

GCS_DATA_SOURCE

HOURS

HTTP_DATA_SOURCE

JOB_NAME

LIST_URL

METADATA

MINUTES

MONTH

NAME

OBJECT_CONDITIONS

OPERATIONS

PATH

PROJECT_ID

SCHEDULE

SCHEDULE_END_DATE

SCHEDULE_START_DATE

SECONDS

SECRET_ACCESS_KEY

START_TIME_OF_DAY

STATUS

STATUS1

TRANSFER_JOB

TRANSFER_JOBS

TRANSFER_JOB_FIELD_MASK

TRANSFER_OPERATIONS

TRANSFER_OPTIONS

TRANSFER_SPEC

YEAR

ALREADY_EXIST_CODE

NEGATIVE_STATUSES

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.log[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TIME_TO_SLEEP_IN_SECONDS = 10[source]
class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferJobsStatus[source]

Class with Google Cloud Transfer jobs statuses.

ENABLED = ENABLED[source]
DISABLED = DISABLED[source]
DELETED = DELETED[source]
class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferOperationStatus[source]

Class with Google Cloud Transfer operations statuses.

IN_PROGRESS = IN_PROGRESS[source]
PAUSED = PAUSED[source]
SUCCESS = SUCCESS[source]
FAILED = FAILED[source]
ABORTED = ABORTED[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ACCESS_KEY_ID = accessKeyId[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXISTING_IN_SINK = overwriteObjectsAlreadyExistingInSink[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_ACCESS_KEY = awsAccessKey[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_S3_DATA_SOURCE = awsS3DataSource[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BODY = body[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BUCKET_NAME = bucketName[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.COUNTERS = counters[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DAY = day[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DESCRIPTION = description[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER = filter[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_JOB_NAMES = job_names[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_PROJECT_ID = project_id[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SINK = gcsDataSink[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SOURCE = gcsDataSource[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HOURS = hours[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HTTP_DATA_SOURCE = httpDataSource[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.JOB_NAME = name[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.LIST_URL = list_url[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.METADATA = metadata[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MINUTES = minutes[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MONTH = month[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NAME = name[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OBJECT_CONDITIONS = object_conditions[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OPERATIONS = operations[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PATH = path[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PROJECT_ID = projectId[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE = schedule[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_END_DATE = scheduleEndDate[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_START_DATE = scheduleStartDate[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECONDS = seconds[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECRET_ACCESS_KEY = secretAccessKey[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.START_TIME_OF_DAY = startTimeOfDay[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS = status[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS1 = status[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB = transfer_job[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOBS = transferJobs[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB_FIELD_MASK = update_transfer_job_field_mask[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPERATIONS = transferOperations[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPTIONS = transfer_options[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_SPEC = transferSpec[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.YEAR = year[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXIST_CODE = 409[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NEGATIVE_STATUSES[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.gen_job_name(job_name)[source]

Adds unique suffix to job name. If suffix already exists, updates it. Suffix — current timestamp

Parameters

job_name (str) --

Returns

job_name with suffix

Return type

str

class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook(api_version='v1', gcp_conn_id='google_cloud_default', delegate_to=None, impersonation_chain=None)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Storage Transfer Service.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

get_conn(self)[source]

Retrieves connection to Google Storage Transfer service.

Returns

Google Storage Transfer service object

Return type

dict

create_transfer_job(self, body)[source]

Creates a transfer job that runs periodically.

Parameters

body (dict) -- (Required) A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body

Returns

transfer job. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob

Return type

dict

get_transfer_job(self, job_name, project_id)[source]

Gets the latest state of a long-running operation in Google Storage Transfer Service.

Parameters

  • job_name (str) -- (Required) Name of the job to be fetched

  • project_id (str) -- (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

Transfer Job

Return type

dict

list_transfer_job(self, request_filter=None, **kwargs)[source]

Lists long-running operations in Google Storage Transfer Service that match the specified filter.

Parameters

request_filter (Optional[dict]) -- (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter

Returns

List of Transfer Jobs

Return type

list[dict]

enable_transfer_job(self, job_name, project_id)[source]

New transfers will be performed based on the schedule.

Parameters

  • job_name (str) -- (Required) Name of the job to be updated

  • project_id (str) -- (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

If successful, TransferJob.

Return type

dict

update_transfer_job(self, job_name, body)[source]

Updates a transfer job that runs periodically.

Parameters
Returns

If successful, TransferJob.

Return type

dict

delete_transfer_job(self, job_name, project_id)[source]

Deletes a transfer job. This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collection. Transfer jobs become eligible for garbage collection 30 days after soft delete.

Parameters

  • job_name (str) -- (Required) Name of the job to be deleted

  • project_id (str) -- (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Return type

None

cancel_transfer_operation(self, operation_name)[source]

Cancels an transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) -- Name of the transfer operation.

Return type

None

get_transfer_operation(self, operation_name)[source]

Gets an transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) -- (Required) Name of the transfer operation.

Returns

transfer operation See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation

Return type

dict

list_transfer_operations(self, request_filter=None, **kwargs)[source]

Gets an transfer operation in Google Storage Transfer Service.

Parameters

request_filter (Optional[dict]) --

(Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter With one additional improvement:

Returns

transfer operation

Return type

list[dict]

pause_transfer_operation(self, operation_name)[source]

Pauses an transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) -- (Required) Name of the transfer operation.

Return type

None

resume_transfer_operation(self, operation_name)[source]

Resumes an transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) -- (Required) Name of the transfer operation.

Return type

None

wait_for_transfer_job(self, job, expected_statuses=None, timeout=None)[source]

Waits until the job reaches the expected state.

Parameters
Return type

None

static operations_contain_expected_statuses(operations, expected_statuses)[source]

Checks whether the operation list has an operation with the expected status, then returns true If it encounters operations in FAILED or ABORTED state throw airflow.exceptions.AirflowException.

Parameters
Returns

If there is an operation with the expected state in the operation list, returns true,

Raises

airflow.exceptions.AirflowException If it encounters operations with a state in the list,

Return type

bool

Was this entry helpful?