airflow.providers.google.cloud.hooks.cloud_storage_transfer_service

This module contains a Google Storage Transfer Service Hook.

Module Contents

Classes

GcpTransferJobsStatus

Google Cloud Transfer job status.

GcpTransferOperationStatus

Google Cloud Transfer operation status.

CloudDataTransferServiceHook

Google Storage Transfer Service functionalities.

CloudDataTransferServiceAsyncHook

Asynchronous hook for Google Storage Transfer Service.

Functions

gen_job_name(job_name)

Add a unique suffix to the job name.

Attributes

log

TIME_TO_SLEEP_IN_SECONDS

ACCESS_KEY_ID

ALREADY_EXISTING_IN_SINK

AWS_ACCESS_KEY

AWS_SECRET_ACCESS_KEY

AWS_S3_DATA_SOURCE

AWS_ROLE_ARN

BODY

BUCKET_NAME

COUNTERS

DAY

DESCRIPTION

FILTER

FILTER_JOB_NAMES

FILTER_PROJECT_ID

GCS_DATA_SINK

GCS_DATA_SOURCE

HOURS

HTTP_DATA_SOURCE

INCLUDE_PREFIXES

JOB_NAME

LIST_URL

METADATA

MINUTES

MONTH

NAME

OBJECT_CONDITIONS

OPERATIONS

OVERWRITE_OBJECTS_ALREADY_EXISTING_IN_SINK

PATH

PROJECT_ID

SCHEDULE

SCHEDULE_END_DATE

SCHEDULE_START_DATE

SECONDS

SECRET_ACCESS_KEY

START_TIME_OF_DAY

STATUS

STATUS1

TRANSFER_JOB

TRANSFER_JOBS

TRANSFER_JOB_FIELD_MASK

TRANSFER_OPERATIONS

TRANSFER_OPTIONS

TRANSFER_SPEC

YEAR

ALREADY_EXIST_CODE

NEGATIVE_STATUSES

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.log[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TIME_TO_SLEEP_IN_SECONDS = 10[source]
class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferJobsStatus[source]

Google Cloud Transfer job status.

ENABLED = 'ENABLED'[source]
DISABLED = 'DISABLED'[source]
DELETED = 'DELETED'[source]
class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GcpTransferOperationStatus[source]

Google Cloud Transfer operation status.

IN_PROGRESS = 'IN_PROGRESS'[source]
PAUSED = 'PAUSED'[source]
SUCCESS = 'SUCCESS'[source]
FAILED = 'FAILED'[source]
ABORTED = 'ABORTED'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ACCESS_KEY_ID = 'accessKeyId'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXISTING_IN_SINK = 'overwriteObjectsAlreadyExistingInSink'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_ACCESS_KEY = 'awsAccessKey'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_SECRET_ACCESS_KEY = 'secretAccessKey'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_S3_DATA_SOURCE = 'awsS3DataSource'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.AWS_ROLE_ARN = 'roleArn'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BODY = 'body'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.BUCKET_NAME = 'bucketName'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.COUNTERS = 'counters'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DAY = 'day'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.DESCRIPTION = 'description'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER = 'filter'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_JOB_NAMES = 'job_names'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.FILTER_PROJECT_ID = 'project_id'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SINK = 'gcsDataSink'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.GCS_DATA_SOURCE = 'gcsDataSource'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HOURS = 'hours'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.HTTP_DATA_SOURCE = 'httpDataSource'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.INCLUDE_PREFIXES = 'includePrefixes'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.JOB_NAME = 'name'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.LIST_URL = 'list_url'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.METADATA = 'metadata'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MINUTES = 'minutes'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.MONTH = 'month'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NAME = 'name'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OBJECT_CONDITIONS = 'object_conditions'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OPERATIONS = 'operations'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.OVERWRITE_OBJECTS_ALREADY_EXISTING_IN_SINK = 'overwriteObjectsAlreadyExistingInSink'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PATH = 'path'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.PROJECT_ID = 'projectId'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE = 'schedule'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_END_DATE = 'scheduleEndDate'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SCHEDULE_START_DATE = 'scheduleStartDate'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECONDS = 'seconds'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.SECRET_ACCESS_KEY = 'secretAccessKey'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.START_TIME_OF_DAY = 'startTimeOfDay'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS = 'status'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.STATUS1 = 'status'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB = 'transfer_job'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOBS = 'transferJobs'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_JOB_FIELD_MASK = 'update_transfer_job_field_mask'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPERATIONS = 'transferOperations'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_OPTIONS = 'transfer_options'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.TRANSFER_SPEC = 'transferSpec'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.YEAR = 'year'[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.ALREADY_EXIST_CODE = 409[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.NEGATIVE_STATUSES[source]
airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.gen_job_name(job_name)[source]

Add a unique suffix to the job name.

Parameters

job_name (str) –

Returns

job_name with suffix

Return type

str

class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook(api_version='v1', gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Google Storage Transfer Service functionalities.

All methods in the hook with project_id in the signature must be called with keyword arguments rather than positional.

get_conn()[source]

Retrieve connection to Google Storage Transfer service.

Returns

Google Storage Transfer service object

Return type

googleapiclient.discovery.Resource

create_transfer_job(body)[source]

Create a transfer job that runs periodically.

Parameters

body (dict) – (Required) The request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body

Returns

The transfer job. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob

Return type

dict

get_transfer_job(job_name, project_id)[source]

Get latest state of a long-running Google Storage Transfer Service job.

Parameters

  • job_name (str) – (Required) Name of the job to be fetched

  • project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

Transfer Job

Return type

dict

list_transfer_job(request_filter=None, **kwargs)[source]

List long-running operations in Google Storage Transfer Service.

A filter can be specified to match only certain entries.

Parameters

request_filter (dict | None) – (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter

Returns

List of Transfer Jobs

Return type

list[dict]

enable_transfer_job(job_name, project_id)[source]

Make new transfers be performed based on the schedule.

Parameters

  • job_name (str) – (Required) Name of the job to be updated

  • project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

If successful, TransferJob.

Return type

dict

update_transfer_job(job_name, body)[source]

Update a transfer job that runs periodically.

Parameters
Returns

If successful, TransferJob.

Return type

dict

delete_transfer_job(job_name, project_id)[source]

Delete a transfer job.

This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collection. Transfer jobs become eligible for garbage collection 30 days after soft delete.

Parameters

  • job_name (str) – (Required) Name of the job to be deleted

  • project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

run_transfer_job(job_name, project_id)[source]

Run Google Storage Transfer Service job.

Parameters

  • job_name (str) – (Required) Name of the job to be fetched

  • project_id (str) – (Optional) the ID of the project that owns the Transfer Job. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

If successful, Operation. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation

Return type

dict

See also

https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/run

cancel_transfer_operation(operation_name)[source]

Cancel a transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) – Name of the transfer operation.

get_transfer_operation(operation_name)[source]

Get a transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) – (Required) Name of the transfer operation.

Returns

transfer operation

Return type

dict

See also

https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation

list_transfer_operations(request_filter=None, **kwargs)[source]

Get a transfer operation in Google Storage Transfer Service.

Parameters

request_filter (dict | None) – (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter With one additional improvement:

Returns

transfer operation

Return type

list[dict]

The project_id parameter is optional if you have a project ID defined in the connection. See: Google Cloud Connection

pause_transfer_operation(operation_name)[source]

Pause a transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) – (Required) Name of the transfer operation.

resume_transfer_operation(operation_name)[source]

Resume a transfer operation in Google Storage Transfer Service.

Parameters

operation_name (str) – (Required) Name of the transfer operation.

wait_for_transfer_job(job, expected_statuses=None, timeout=None)[source]

Wait until the job reaches the expected state.

Parameters
static operations_contain_expected_statuses(operations, expected_statuses)[source]

Check whether an operation exists with the expected status.

Parameters
Returns

If there is an operation with the expected state in the operation list, returns true,

Raises

AirflowException – If it encounters operations with state FAILED or ABORTED in the list.

Return type

bool

class airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceAsyncHook(project_id=PROVIDE_PROJECT_ID, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseAsyncHook

Asynchronous hook for Google Storage Transfer Service.

sync_hook_class[source]
async get_conn()[source]

Return async connection to the Storage Transfer Service.

Returns

Google Storage Transfer asynchronous client.

Return type

google.cloud.storage_transfer_v1.StorageTransferServiceAsyncClient

async get_jobs(job_names)[source]

Get the latest state of a long-running operations in Google Storage Transfer Service.

Parameters

job_names (list[str]) – (Required) List of names of the jobs to be fetched.

Returns

Object that yields Transfer jobs.

Return type

google.cloud.storage_transfer_v1.services.storage_transfer_service.pagers.ListTransferJobsAsyncPager

async get_latest_operation(job)[source]

Get the latest operation of the given TransferJob instance.

Parameters

job (google.cloud.storage_transfer_v1.TransferJob) – Transfer job instance.

Returns

The latest job operation.

Return type

proto.Message | None

Was this entry helpful?