`airflow.providers.google.cloud.hooks.cloud_storage_transfer_service`¶

This module contains a Google Storage Transfer Service Hook.

Module Contents¶

Classes¶

`GcpTransferJobsStatus`	Google Cloud Transfer job status.
`GcpTransferOperationStatus`	Google Cloud Transfer operation status.
`CloudDataTransferServiceHook`	Google Storage Transfer Service functionalities.

Functions¶

gen_job_name(job_name)

Add a unique suffix to the job name.

Attributes¶

`log`
`TIME_TO_SLEEP_IN_SECONDS`
`ACCESS_KEY_ID`
`ALREADY_EXISTING_IN_SINK`
`AWS_ACCESS_KEY`
`AWS_S3_DATA_SOURCE`
`BODY`
`BUCKET_NAME`
`COUNTERS`
`DAY`
`DESCRIPTION`
`FILTER`
`FILTER_JOB_NAMES`
`FILTER_PROJECT_ID`
`GCS_DATA_SINK`
`GCS_DATA_SOURCE`
`HOURS`
`HTTP_DATA_SOURCE`
`JOB_NAME`
`LIST_URL`
`METADATA`
`MINUTES`
`MONTH`
`NAME`
`OBJECT_CONDITIONS`
`OPERATIONS`
`PATH`
`PROJECT_ID`
`SCHEDULE`
`SCHEDULE_END_DATE`
`SCHEDULE_START_DATE`
`SECONDS`
`SECRET_ACCESS_KEY`
`START_TIME_OF_DAY`
`STATUS`
`STATUS1`
`TRANSFER_JOB`
`TRANSFER_JOBS`
`TRANSFER_JOB_FIELD_MASK`
`TRANSFER_OPERATIONS`
`TRANSFER_OPTIONS`
`TRANSFER_SPEC`
`YEAR`
`ALREADY_EXIST_CODE`
`NEGATIVE_STATUSES`

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.log[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TIME_TO_SLEEP_IN_SECONDS = 10[source]¶

class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferJobsStatus[source]¶

Google Cloud Transfer job status.

ENABLED = 'ENABLED'[source]¶

DISABLED = 'DISABLED'[source]¶

DELETED = 'DELETED'[source]¶

class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferOperationStatus[source]¶

Google Cloud Transfer operation status.

IN_PROGRESS = 'IN_PROGRESS'[source]¶

PAUSED = 'PAUSED'[source]¶

SUCCESS = 'SUCCESS'[source]¶

FAILED = 'FAILED'[source]¶

ABORTED = 'ABORTED'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ACCESS_KEY_ID = 'accessKeyId'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXISTING_IN_SINK = 'overwriteObjectsAlreadyExistingInSink'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_ACCESS_KEY = 'awsAccessKey'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_S3_DATA_SOURCE = 'awsS3DataSource'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BODY = 'body'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BUCKET_NAME = 'bucketName'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.COUNTERS = 'counters'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DAY = 'day'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DESCRIPTION = 'description'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER = 'filter'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_JOB_NAMES = 'job_names'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_PROJECT_ID = 'project_id'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SINK = 'gcsDataSink'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SOURCE = 'gcsDataSource'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HOURS = 'hours'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HTTP_DATA_SOURCE = 'httpDataSource'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.JOB_NAME = 'name'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.LIST_URL = 'list_url'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.METADATA = 'metadata'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MINUTES = 'minutes'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MONTH = 'month'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NAME = 'name'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OBJECT_CONDITIONS = 'object_conditions'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OPERATIONS = 'operations'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PATH = 'path'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PROJECT_ID = 'projectId'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE = 'schedule'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_END_DATE = 'scheduleEndDate'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_START_DATE = 'scheduleStartDate'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECONDS = 'seconds'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECRET_ACCESS_KEY = 'secretAccessKey'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.START_TIME_OF_DAY = 'startTimeOfDay'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS = 'status'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS1 = 'status'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB = 'transfer_job'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOBS = 'transferJobs'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB_FIELD_MASK = 'update_transfer_job_field_mask'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPERATIONS = 'transferOperations'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPTIONS = 'transfer_options'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_SPEC = 'transferSpec'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.YEAR = 'year'[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXIST_CODE = 409[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NEGATIVE_STATUSES[source]¶

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.gen_job_name(job_name)[source]¶

Add a unique suffix to the job name.

Parameters: job_name (str) –
Returns: job_name with suffix
Return type: str

class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook(api_version='v1', gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Google Storage Transfer Service functionalities.

All methods in the hook with project_id in the signature must be called with keyword arguments rather than positional.

get_conn()[source]¶

Retrieve connection to Google Storage Transfer service.

Returns: Google Storage Transfer service object
Return type: googleapiclient.discovery.Resource

create_transfer_job(body)[source]¶

Create a transfer job that runs periodically.

Parameters: body (dict) – (Required) The request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
Returns: The transfer job. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
Return type: dict

get_transfer_job(job_name, project_id)[source]¶

Get latest state of a long-running Google Storage Transfer Service job.

Parameters

job_name (str) – (Required) Name of the job to be fetched
project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

Transfer Job

Return type

dict

list_transfer_job(request_filter=None, **kwargs)[source]¶

List long-running operations in Google Storage Transfer Service.

A filter can be specified to match only certain entries.

Parameters: request_filter (dict | None) – (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
Returns: List of Transfer Jobs
Return type: list[dict]

enable_transfer_job(job_name, project_id)[source]¶

Make new transfers be performed based on the schedule.

Parameters

job_name (str) – (Required) Name of the job to be updated
project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

If successful, TransferJob.

Return type

dict

update_transfer_job(job_name, body)[source]¶

Update a transfer job that runs periodically.

Parameters

job_name (str) – (Required) Name of the job to be updated
body (dict) – A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body

Returns

If successful, TransferJob.

Return type

dict

delete_transfer_job(job_name, project_id)[source]¶

Delete a transfer job.

This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collection. Transfer jobs become eligible for garbage collection 30 days after soft delete.

Parameters

job_name (str) – (Required) Name of the job to be deleted
project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

cancel_transfer_operation(operation_name)[source]¶

Cancel a transfer operation in Google Storage Transfer Service.

Parameters: operation_name (str) – Name of the transfer operation.

get_transfer_operation(operation_name)[source]¶

Get a transfer operation in Google Storage Transfer Service.

Parameters: operation_name (str) – (Required) Name of the transfer operation.
Returns: transfer operation
Return type: dict

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service¶

Module Contents¶

Classes¶

Functions¶

Attributes¶

`airflow.providers.google.cloud.hooks.cloud_storage_transfer_service`¶