airflow.providers.google.cloud.sensors.dataproc

This module contains a Dataproc Job sensor.

Module Contents

Classes

DataprocJobSensor

Check for the state of a previously submitted Dataproc job.

DataprocBatchSensor

Check for the state of batch.

class airflow.providers.google.cloud.sensors.dataproc.DataprocJobSensor(*, dataproc_job_id, region, project_id=None, gcp_conn_id='google_cloud_default', wait_timeout=None, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Check for the state of a previously submitted Dataproc job.

Parameters
  • dataproc_job_id (str) – The Dataproc job ID to poll. (templated)

  • region (str) – Required. The Cloud Dataproc region in which to handle the request. (templated)

  • project_id (str | None) – The ID of the google cloud project in which to create the cluster. (templated)

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud Platform.

  • wait_timeout (int | None) – How many seconds wait for job to be ready.

template_fields: Sequence[str] = ('project_id', 'region', 'dataproc_job_id')[source]
ui_color = '#f0eee4'[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

poke(context)[source]

Override when deriving this class.

class airflow.providers.google.cloud.sensors.dataproc.DataprocBatchSensor(*, batch_id, region, project_id=None, gcp_conn_id='google_cloud_default', wait_timeout=None, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Check for the state of batch.

Parameters
  • batch_id (str) – The Dataproc batch ID to poll. (templated)

  • region (str) – Required. The Cloud Dataproc region in which to handle the request. (templated)

  • project_id (str | None) – The ID of the google cloud project in which to create the cluster. (templated)

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud Platform.

  • wait_timeout (int | None) – How many seconds wait for job to be ready.

template_fields: Sequence[str] = ('project_id', 'region', 'batch_id')[source]
ui_color = '#f0eee4'[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

poke(context)[source]

Override when deriving this class.

Was this entry helpful?