airflow.providers.google.cloud.sensors.cloud_composer

This module contains a Cloud Composer sensor.

Module Contents

Classes

CloudComposerEnvironmentSensor

Check the status of the Cloud Composer Environment task.

CloudComposerDAGRunSensor

Check if a DAG run has completed.
class airflow.providers.google.cloud.sensors.cloud_composer.CloudComposerEnvironmentSensor(*, project_id, region, operation_name, gcp_conn_id='google_cloud_default', impersonation_chain=None, pooling_period_seconds=30, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Check the status of the Cloud Composer Environment task.

This Sensor is deprecated. You can achieve the same functionality by using Cloud Composer Operators CloudComposerCreateEnvironmentOperator, CloudComposerDeleteEnvironmentOperator and CloudComposerUpdateEnvironmentOperator in deferrable or non-deferrable mode, since every operator gives user a possibility to wait (asynchronously or synchronously) until Operation will be finished.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • operation_name (str) – The name of the operation resource

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

  • pooling_period_seconds (int) – Optional: Control the rate of the poll for the result of deferrable run.

execute(context)[source]

Airflow runs this method on the worker and defers using the trigger.

execute_complete(context, event=None)[source]

Act as a callback for when the trigger fires - returns immediately.

Relies on trigger to throw an exception, otherwise it assumes execution was successful.

class airflow.providers.google.cloud.sensors.cloud_composer.CloudComposerDAGRunSensor(*, project_id, region, environment_id, composer_dag_id, allowed_states=None, execution_range=None, gcp_conn_id='google_cloud_default', impersonation_chain=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), poll_interval=10, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Check if a DAG run has completed.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – The name of the Composer environment.

  • composer_dag_id (str) – The ID of executable DAG.

  • allowed_states (Iterable[str] | None) – Iterable of allowed states, default is ['success'].

  • execution_range (datetime.timedelta | list[datetime.datetime] | None) – execution DAGs time range. Sensor checks DAGs states only for DAGs which were started in this time range. For yesterday, use [positive!] datetime.timedelta(days=1). For future, use [negative!] datetime.timedelta(days=-1). For specific time, use list of datetimes [datetime(2024,3,22,11,0,0), datetime(2024,3,22,12,0,0)]. Or [datetime(2024,3,22,0,0,0)] in this case sensor will check for states from specific time in the past till current time execution. Default value datetime.timedelta(days=1).

  • gcp_conn_id (str) – The connection ID to use when fetching connection info.

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

  • poll_interval (int) – Optional: Control the rate of the poll for the result of deferrable run.

  • deferrable (bool) – Run sensor in deferrable mode.

template_fields = ('project_id', 'region', 'environment_id', 'composer_dag_id', 'impersonation_chain')[source]
poke(context)[source]

Override when deriving this class.

execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event)[source]

Was this entry helpful?