airflow.providers.apache.beam.triggers.beam

Module Contents

Classes

BeamPipelineTrigger

Trigger to perform checking the pipeline status until it reaches terminate state.
class airflow.providers.apache.beam.triggers.beam.BeamPipelineTrigger(variables, py_file, py_options=None, py_interpreter='python3', py_requirements=None, py_system_site_packages=False, runner='DirectRunner')[source]

Bases: airflow.triggers.base.BaseTrigger

Trigger to perform checking the pipeline status until it reaches terminate state.

Parameters

  • variables (dict) – Variables passed to the pipeline.

  • py_file (str) – Path to the python file to execute.

  • py_options (list[str] | None) – Additional options.

  • py_interpreter (str) – Python version of the Apache Beam pipeline. If None, this defaults to the python3. To track python versions supported by beam and related issues check: https://issues.apache.org/jira/browse/BEAM-1251

  • py_requirements (list[str] | None) –

    Additional python package(s) to install. If a value is passed to this parameter, a new virtual environment has been created with additional packages installed.

    You could also install the apache-beam package if it is not installed on your system, or you want to use a different version.

  • py_system_site_packages (bool) –

    Whether to include system_site_packages in your virtualenv. See virtualenv documentation for more information.

    This option is only relevant if the py_requirements parameter is not None.

  • runner (str) – Runner on which pipeline will be run. By default, “DirectRunner” is being used. Other possible options: DataflowRunner, SparkRunner, FlinkRunner, PortableRunner. See: BeamRunnerType See: https://beam.apache.org/documentation/runners/capability-matrix/

serialize()[source]

Serialize BeamPipelineTrigger arguments and classpath.

async run()[source]

Get current pipeline status and yields a TriggerEvent.

Was this entry helpful?