• Home
• airflow.providers.google
• airflow.providers.google.cloud
• airflow.providers.google.cloud.triggers
• airflow.providers.google.cloud.triggers.dataflow

airflow.providers.google.cloud.triggers.dataflow

Module Contents

Classes

TemplateJobStartTrigger	Dataflow trigger to check if templated job has been finished.
DataflowJobStatusTrigger	Trigger that checks for metrics associated with a Dataflow job.
DataflowJobMetricsTrigger	Trigger that checks for metrics associated with a Dataflow job.
DataflowJobAutoScalingEventTrigger	Trigger that checks for autoscaling events associated with a Dataflow job.
DataflowJobMessagesTrigger	Trigger that checks for job messages associated with a Dataflow job.

Attributes

DEFAULT_DATAFLOW_LOCATION

airflow.providers.google.cloud.triggers.dataflow.DEFAULT_DATAFLOW_LOCATION = 'us-central1'

class airflow.providers.google.cloud.triggers.dataflow.TemplateJobStartTrigger(job_id, project_id, location=DEFAULT_DATAFLOW_LOCATION, gcp_conn_id='google_cloud_default', poll_sleep=10, impersonation_chain=None, cancel_timeout=5 * 60)

Bases: airflow.triggers.base.BaseTrigger

Dataflow trigger to check if templated job has been finished.

Parameters

• project_id (str | None) – Required. the Google Cloud project ID in which the job was started.

• job_id (str) – Required. ID of the job.

• location (str) – Optional. the location where job is executed. If set to None then the value of DEFAULT_DATAFLOW_LOCATION will be used

• gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

• impersonation_chain (str | Sequence[str] | None) – Optional. Service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

• cancel_timeout (int | None) – Optional. How long (in seconds) operator should wait for the pipeline to be successfully cancelled when task is being killed.

serialize()

Serialize class arguments and classpath.

async run()

Fetch job status or yield certain Events.

Main loop of the class in where it is fetching the job status and yields certain Event.

If the job has status success then it yields TriggerEvent with success status, if job has status failed - with error status. In any other case Trigger will wait for specified amount of time stored in self.poll_sleep variable.

class airflow.providers.google.cloud.triggers.dataflow.DataflowJobStatusTrigger(job_id, expected_statuses, project_id, location=DEFAULT_DATAFLOW_LOCATION, gcp_conn_id='google_cloud_default', poll_sleep=10, impersonation_chain=None)

Bases: airflow.triggers.base.BaseTrigger

Trigger that checks for metrics associated with a Dataflow job.

Parameters

• job_id (str) – Required. ID of the job.

• expected_statuses (set[str]) – The expected state(s) of the operation. See: https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.JobState

• project_id (str | None) – Required. The Google Cloud project ID in which the job was started.

• location (str) – Optional. The location where the job is executed. If set to None then the value of DEFAULT_DATAFLOW_LOCATION will be used.

• gcp_conn_id (str) – The connection ID to use for connecting to Google Cloud.

• poll_sleep (int) – Time (seconds) to wait between two consecutive calls to check the job.

• impersonation_chain (str | Sequence[str] | None) – Optional. Service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

serialize()

Serialize class arguments and classpath.

async run()

Loop until the job reaches an expected or terminal state.

Yields a TriggerEvent with success status, if the client returns an expected job status.

Yields a TriggerEvent with error status, if the client returns an unexpected terminal job status or any exception is raised while looping.

In any other case the Trigger will wait for a specified amount of time stored in self.poll_sleep variable.

async_hook()

class airflow.providers.google.cloud.triggers.dataflow.DataflowJobMetricsTrigger(job_id, project_id, location=DEFAULT_DATAFLOW_LOCATION, gcp_conn_id='google_cloud_default', poll_sleep=10, impersonation_chain=None, fail_on_terminal_state=True)

Bases: airflow.triggers.base.BaseTrigger

Trigger that checks for metrics associated with a Dataflow job.

Parameters

• job_id (str) – Required. ID of the job.

• project_id (str | None) – Required. The Google Cloud project ID in which the job was started.

• location (str) – Optional. The location where the job is executed. If set to None then the value of DEFAULT_DATAFLOW_LOCATION will be used.

• gcp_conn_id (str) – The connection ID to use for connecting to Google Cloud.

• poll_sleep (int) – Time (seconds) to wait between two consecutive calls to check the job.

• impersonation_chain (str | Sequence[str] | None) – Optional. Service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

• fail_on_terminal_state (bool) – If set to True the trigger will yield a TriggerEvent with error status if the job reaches a terminal state.

serialize()

Serialize class arguments and classpath.

async run()

Loop until a terminal job status or any job metrics are returned.

Yields a TriggerEvent with success status, if the client returns any job metrics and fail_on_terminal_state attribute is False.

Yields a TriggerEvent with error status, if the client returns a job status with a terminal state value and fail_on_terminal_state attribute is True.

Yields a TriggerEvent with error status, if any exception is raised while looping.

In any other case the Trigger will wait for a specified amount of time stored in self.poll_sleep variable.

async get_job_metrics()

Wait for the Dataflow client response and then return it in a serialized list.

async_hook()

class airflow.providers.google.cloud.triggers.dataflow.DataflowJobAutoScalingEventTrigger(job_id, project_id, location=DEFAULT_DATAFLOW_LOCATION, gcp_conn_id='google_cloud_default', poll_sleep=10, impersonation_chain=None, fail_on_terminal_state=True)

Bases: airflow.triggers.base.BaseTrigger

Trigger that checks for autoscaling events associated with a Dataflow job.

Parameters

• job_id (str) – Required. ID of the job.

• project_id (str | None) – Required. The Google Cloud project ID in which the job was started.

• location (str) – Optional. The location where the job is executed. If set to None then the value of DEFAULT_DATAFLOW_LOCATION will be used.

• gcp_conn_id (str) – The connection ID to use for connecting to Google Cloud.

• poll_sleep (int) – Time (seconds) to wait between two consecutive calls to check the job.

• impersonation_chain (str | Sequence[str] | None) – Optional. Service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

• fail_on_terminal_state (bool) – If set to True the trigger will yield a TriggerEvent with error status if the job reaches a terminal state.

serialize()

Serialize class arguments and classpath.

async run()

Loop until a terminal job status or any autoscaling events are returned.

Yields a TriggerEvent with success status, if the client returns any autoscaling events and fail_on_terminal_state attribute is False.

Yields a TriggerEvent with error status, if the client returns a job status with a terminal state value and fail_on_terminal_state attribute is True.

Yields a TriggerEvent with error status, if any exception is raised while looping.

In any other case the Trigger will wait for a specified amount of time stored in self.poll_sleep variable.

async list_job_autoscaling_events()

Wait for the Dataflow client response and then return it in a serialized list.

async_hook()

class airflow.providers.google.cloud.triggers.dataflow.DataflowJobMessagesTrigger(job_id, project_id, location=DEFAULT_DATAFLOW_LOCATION, gcp_conn_id='google_cloud_default', poll_sleep=10, impersonation_chain=None, fail_on_terminal_state=True)

Bases: airflow.triggers.base.BaseTrigger

Trigger that checks for job messages associated with a Dataflow job.

Parameters

• job_id (str) – Required. ID of the job.

• project_id (str | None) – Required. The Google Cloud project ID in which the job was started.

• location (str) – Optional. The location where the job is executed. If set to None then the value of DEFAULT_DATAFLOW_LOCATION will be used.

• gcp_conn_id (str) – The connection ID to use for connecting to Google Cloud.

• poll_sleep (int) – Time (seconds) to wait between two consecutive calls to check the job.

• impersonation_chain (str | Sequence[str] | None) – Optional. Service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

• fail_on_terminal_state (bool) – If set to True the trigger will yield a TriggerEvent with error status if the job reaches a terminal state.

serialize()

Serialize class arguments and classpath.

async run()

Loop until a terminal job status or any job messages are returned.

Yields a TriggerEvent with success status, if the client returns any job messages and fail_on_terminal_state attribute is False.

Yields a TriggerEvent with error status, if the client returns a job status with a terminal state value and fail_on_terminal_state attribute is True.

Yields a TriggerEvent with error status, if any exception is raised while looping.

In any other case the Trigger will wait for a specified amount of time stored in self.poll_sleep variable.

async list_job_messages()

Wait for the Dataflow client response and then return it in a serialized list.

async_hook()

Previous Next

Was this entry helpful?