:mod:`airflow.contrib.hooks.gcp_transfer_hook`
==============================================
.. py:module:: airflow.contrib.hooks.gcp_transfer_hook
Module Contents
---------------
.. data:: TIME_TO_SLEEP_IN_SECONDS
:annotation: = 10
.. py:class:: GcpTransferJobsStatus
.. attribute:: ENABLED
:annotation: = ENABLED
.. attribute:: DISABLED
:annotation: = DISABLED
.. attribute:: DELETED
:annotation: = DELETED
.. py:class:: GcpTransferOperationStatus
.. attribute:: IN_PROGRESS
:annotation: = IN_PROGRESS
.. attribute:: PAUSED
:annotation: = PAUSED
.. attribute:: SUCCESS
:annotation: = SUCCESS
.. attribute:: FAILED
:annotation: = FAILED
.. attribute:: ABORTED
:annotation: = ABORTED
.. data:: ACCESS_KEY_ID
:annotation: = accessKeyId
.. data:: ALREADY_EXISTING_IN_SINK
:annotation: = overwriteObjectsAlreadyExistingInSink
.. data:: AWS_ACCESS_KEY
:annotation: = awsAccessKey
.. data:: AWS_S3_DATA_SOURCE
:annotation: = awsS3DataSource
.. data:: BODY
:annotation: = body
.. data:: BUCKET_NAME
:annotation: = bucketName
.. data:: DAY
:annotation: = day
.. data:: DESCRIPTION
:annotation: = description
.. data:: FILTER
:annotation: = filter
.. data:: FILTER_JOB_NAMES
:annotation: = job_names
.. data:: FILTER_PROJECT_ID
:annotation: = project_id
.. data:: GCS_DATA_SINK
:annotation: = gcsDataSink
.. data:: GCS_DATA_SOURCE
:annotation: = gcsDataSource
.. data:: HOURS
:annotation: = hours
.. data:: HTTP_DATA_SOURCE
:annotation: = httpDataSource
.. data:: LIST_URL
:annotation: = list_url
.. data:: METADATA
:annotation: = metadata
.. data:: MINUTES
:annotation: = minutes
.. data:: MONTH
:annotation: = month
.. data:: NAME
:annotation: = name
.. data:: OBJECT_CONDITIONS
:annotation: = object_conditions
.. data:: OPERATIONS
:annotation: = operations
.. data:: PROJECT_ID
:annotation: = projectId
.. data:: SCHEDULE
:annotation: = schedule
.. data:: SCHEDULE_END_DATE
:annotation: = scheduleEndDate
.. data:: SCHEDULE_START_DATE
:annotation: = scheduleStartDate
.. data:: SECONDS
:annotation: = seconds
.. data:: SECRET_ACCESS_KEY
:annotation: = secretAccessKey
.. data:: START_TIME_OF_DAY
:annotation: = startTimeOfDay
.. data:: STATUS
:annotation: = status
.. data:: STATUS1
:annotation: = status
.. data:: TRANSFER_JOB
:annotation: = transfer_job
.. data:: TRANSFER_JOB_FIELD_MASK
:annotation: = update_transfer_job_field_mask
.. data:: TRANSFER_JOBS
:annotation: = transferJobs
.. data:: TRANSFER_OPERATIONS
:annotation: = transferOperations
.. data:: TRANSFER_OPTIONS
:annotation: = transfer_options
.. data:: TRANSFER_SPEC
:annotation: = transferSpec
.. data:: YEAR
:annotation: = year
.. data:: NEGATIVE_STATUSES
.. py:class:: GCPTransferServiceHook(api_version='v1', gcp_conn_id='google_cloud_default', delegate_to=None)
Bases: :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`
Hook for Google Storage Transfer Service.
.. attribute:: _conn
.. method:: get_conn(self)
Retrieves connection to Google Storage Transfer service.
:return: Google Storage Transfer service object
:rtype: dict
.. method:: create_transfer_job(self, body)
Creates a transfer job that runs periodically.
:param body: (Required) A request body, as described in
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
:type body: dict
:return: transfer job.
See:
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
:rtype: dict
.. method:: get_transfer_job(self, job_name, project_id=None)
Gets the latest state of a long-running operation in Google Storage
Transfer Service.
:param job_name: (Required) Name of the job to be fetched
:type job_name: str
:param project_id: (Optional) the ID of the project that owns the Transfer
Job. If set to None or missing, the default project_id from the GCP
connection is used.
:type project_id: str
:return: Transfer Job
:rtype: dict
.. method:: list_transfer_job(self, filter)
Lists long-running operations in Google Storage Transfer
Service that match the specified filter.
:param filter: (Required) A request filter, as described in
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
:type filter: dict
:return: List of Transfer Jobs
:rtype: list[dict]
.. method:: update_transfer_job(self, job_name, body)
Updates a transfer job that runs periodically.
:param job_name: (Required) Name of the job to be updated
:type job_name: str
:param body: A request body, as described in
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
:type body: dict
:return: If successful, TransferJob.
:rtype: dict
.. method:: delete_transfer_job(self, job_name, project_id)
Deletes a transfer job. This is a soft delete. After a transfer job is
deleted, the job and all the transfer executions are subject to garbage
collection. Transfer jobs become eligible for garbage collection
30 days after soft delete.
:param job_name: (Required) Name of the job to be deleted
:type job_name: str
:param project_id: (Optional) the ID of the project that owns the Transfer
Job. If set to None or missing, the default project_id from the GCP
connection is used.
:type project_id: str
:rtype: None
.. method:: cancel_transfer_operation(self, operation_name)
Cancels an transfer operation in Google Storage Transfer Service.
:param operation_name: Name of the transfer operation.
:type operation_name: str
:rtype: None
.. method:: get_transfer_operation(self, operation_name)
Gets an transfer operation in Google Storage Transfer Service.
:param operation_name: (Required) Name of the transfer operation.
:type operation_name: str
:return: transfer operation
See:
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation
:rtype: dict
.. method:: list_transfer_operations(self, filter)
Gets an transfer operation in Google Storage Transfer Service.
:param filter: (Required) A request filter, as described in
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
With one additional improvement:
* project_id is optional if you have a project id defined
in the connection
See: :ref:`howto/connection:gcp`
:type filter: dict
:return: transfer operation
:rtype: list[dict]
.. method:: pause_transfer_operation(self, operation_name)
Pauses an transfer operation in Google Storage Transfer Service.
:param operation_name: (Required) Name of the transfer operation.
:type operation_name: str
:rtype: None
.. method:: resume_transfer_operation(self, operation_name)
Resumes an transfer operation in Google Storage Transfer Service.
:param operation_name: (Required) Name of the transfer operation.
:type operation_name: str
:rtype: None
.. method:: wait_for_transfer_job(self, job, expected_statuses=(GcpTransferOperationStatus.SUCCESS, ), timeout=60)
Waits until the job reaches the expected state.
:param job: Transfer job
See:
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
:type job: dict
:param expected_statuses: State that is expected
See:
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
:type expected_statuses: set[str]
:param timeout:
:type timeout: time in which the operation must end in seconds
:rtype: None
.. method:: _inject_project_id(self, body, param_name, target_key)
.. staticmethod:: operations_contain_expected_statuses(operations, expected_statuses)
Checks whether the operation list has an operation with the
expected status, then returns true
If it encounters operations in FAILED or ABORTED state
throw :class:`airflow.exceptions.AirflowException`.
:param operations: (Required) List of transfer operations to check.
:type operations: list[dict]
:param expected_statuses: (Required) status that is expected
See:
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
:type expected_statuses: set[str]
:return: If there is an operation with the expected state
in the operation list, returns true,
:raises: airflow.exceptions.AirflowException If it encounters operations
with a state in the list,
:rtype: bool