airflow.providers.google.cloud.hooks.datapipeline
¶
This module contains a Google Data Pipelines Hook.
Module Contents¶
Classes¶
Hook for Google Data Pipelines. |
Attributes¶
- airflow.providers.google.cloud.hooks.datapipeline.DEFAULT_DATAPIPELINE_LOCATION = 'us-central1'[source]¶
- class airflow.providers.google.cloud.hooks.datapipeline.DataPipelineHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶
Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseHook
Hook for Google Data Pipelines.
All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.
- create_data_pipeline(body, project_id, location=DEFAULT_DATAPIPELINE_LOCATION)[source]¶
Create a new Data Pipelines instance from the Data Pipelines API.
- Parameters
body (dict) – The request body (contains instance of Pipeline). See: https://cloud.google.com/dataflow/docs/reference/data-pipelines/rest/v1/projects.locations.pipelines/create#request-body
project_id (str) – The ID of the GCP project that owns the job.
location (str) – The location to direct the Data Pipelines instance to (for example us-central1).
Returns the created Data Pipelines instance in JSON representation.
- run_data_pipeline(data_pipeline_name, project_id, location=DEFAULT_DATAPIPELINE_LOCATION)[source]¶
Run a Data Pipelines Instance using the Data Pipelines API.
- Parameters
data_pipeline_name (str) – The display name of the pipeline. In example projects/PROJECT_ID/locations/LOCATION_ID/pipelines/PIPELINE_ID it would be the PIPELINE_ID.
project_id (str) – The ID of the GCP project that owns the job.
location (str) – The location to direct the Data Pipelines instance to (for example us-central1).
Returns the created Job in JSON representation.