airflow.providers.google.cloud.hooks.dataform

Module Contents

Classes

DataformHook

Hook for Google Cloud DataForm APIs.
class airflow.providers.google.cloud.hooks.dataform.DataformHook(gcp_conn_id='google_cloud_default', delegate_to=None, impersonation_chain=None)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud DataForm APIs.

get_dataform_client()[source]

Retrieves client library object that allow access to Cloud Dataform service.

wait_for_workflow_invocation(workflow_invocation_id, repository_id, project_id, region, wait_time=10, timeout=None)[source]

Helper method which polls a job to check if it finishes.

Parameters

  • workflow_invocation_id (str) – Id of the Workflow Invocation

  • repository_id (str) – Id of the Dataform repository

  • project_id (str) – Required. The ID of the Google Cloud project the cluster belongs to.

  • region (str) – Required. The Cloud Dataproc region in which to handle the request.

  • wait_time (int) – Number of seconds between checks

  • timeout (Optional[int]) – How many seconds wait for job to be ready. Used only if asynchronous is False

create_compilation_result(project_id, region, repository_id, compilation_result, retry=DEFAULT, timeout=None, metadata=())[source]

Creates a new CompilationResult in a given project and location.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.

  • compilation_result (Union[google.cloud.dataform_v1beta1.types.CompilationResult, Dict]) – Required. The compilation result to create.

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – Designation of what errors, if any, should be retried.

  • timeout (Optional[float]) – The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) – Strings which should be sent along with the request as metadata.

get_compilation_result(project_id, region, repository_id, compilation_result_id, retry=DEFAULT, timeout=None, metadata=())[source]

Fetches a single CompilationResult.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.

  • compilation_result_id (str) – The Id of the Dataform Compilation Result

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – Designation of what errors, if any, should be retried.

  • timeout (Optional[float]) – The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) – Strings which should be sent along with the request as metadata.

create_workflow_invocation(project_id, region, repository_id, workflow_invocation, retry=DEFAULT, timeout=None, metadata=())[source]

Creates a new WorkflowInvocation in a given Repository.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.

  • workflow_invocation (Union[google.cloud.dataform_v1beta1.types.WorkflowInvocation, Dict]) – Required. The workflow invocation resource to create.

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – Designation of what errors, if any, should be retried.

  • timeout (Optional[float]) – The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) – Strings which should be sent along with the request as metadata.

get_workflow_invocation(project_id, region, repository_id, workflow_invocation_id, retry=DEFAULT, timeout=None, metadata=())[source]

Fetches a single WorkflowInvocation.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.

  • workflow_invocation_id (str) – Required. The workflow invocation resource’s id.

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – Designation of what errors, if any, should be retried.

  • timeout (Optional[float]) – The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) – Strings which should be sent along with the request as metadata.

cancel_workflow_invocation(project_id, region, repository_id, workflow_invocation_id, retry=DEFAULT, timeout=None, metadata=())[source]

Requests cancellation of a running WorkflowInvocation.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the task belongs to.

  • repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.

  • workflow_invocation_id (str) – Required. The workflow invocation resource’s id.

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – Designation of what errors, if any, should be retried.

  • timeout (Optional[float]) – The timeout for this request.

  • metadata (Sequence[Tuple[str, str]]) – Strings which should be sent along with the request as metadata.

Was this entry helpful?