airflow.providers.google.cloud.hooks.datapipeline

This module contains a Google Data Pipelines Hook.

Module Contents

Classes

DataPipelineHook

Hook for Google Data Pipelines.

Attributes

DEFAULT_DATAPIPELINE_LOCATION

airflow.providers.google.cloud.hooks.datapipeline.DEFAULT_DATAPIPELINE_LOCATION = 'us-central1'[source]
class airflow.providers.google.cloud.hooks.datapipeline.DataPipelineHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Data Pipelines.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

get_conn()[source]

Return a Google Cloud Data Pipelines service object.

create_data_pipeline(body, project_id, location=DEFAULT_DATAPIPELINE_LOCATION)[source]

Create a new Data Pipelines instance from the Data Pipelines API.

Parameters

Returns the created Data Pipelines instance in JSON representation.

run_data_pipeline(data_pipeline_name, project_id, location=DEFAULT_DATAPIPELINE_LOCATION)[source]

Run a Data Pipelines Instance using the Data Pipelines API.

Parameters

  • data_pipeline_name (str) – The display name of the pipeline. In example projects/PROJECT_ID/locations/LOCATION_ID/pipelines/PIPELINE_ID it would be the PIPELINE_ID.

  • project_id (str) – The ID of the GCP project that owns the job.

  • location (str) – The location to direct the Data Pipelines instance to (for example us-central1).

Returns the created Job in JSON representation.

static build_parent_name(project_id, location)[source]

Was this entry helpful?