airflow.providers.apache.livy.triggers.livy

This module contains the Apache Livy Trigger.

Module Contents

Classes

LivyTrigger

Check for the state of a previously submitted job with batch_id.
class airflow.providers.apache.livy.triggers.livy.LivyTrigger(batch_id, spark_params, livy_conn_id='livy_default', polling_interval=0, extra_options=None, extra_headers=None, livy_hook_async=None)[source]

Bases: airflow.triggers.base.BaseTrigger

Check for the state of a previously submitted job with batch_id.

Parameters

  • batch_id (int | str) – Batch job id

  • spark_params (dict[Any, Any]) – Spark parameters; for example, spark_params = {“file”: “test/pi.py”, “class_name”: “org.apache.spark.examples.SparkPi”, “args”: [“/usr/lib/spark/bin/run-example”, “SparkPi”, “10”],”jars”: “command-runner.jar”, “driver_cores”: 1, “executor_cores”: 4,”num_executors”: 1}

  • livy_conn_id (str) – reference to a pre-defined Livy Connection.

  • polling_interval (int) – time in seconds between polling for job completion. If poll_interval=0, in that case return the batch_id and if polling_interval > 0, poll the livy job for termination in the polling interval defined.

  • extra_options (dict[str, Any] | None) – A dictionary of options, where key is string and value depends on the option that’s being modified.

  • extra_headers (dict[str, Any] | None) – A dictionary of headers passed to the HTTP request to livy.

  • livy_hook_async (airflow.providers.apache.livy.hooks.livy.LivyAsyncHook | None) – LivyAsyncHook object

serialize()[source]

Serialize LivyTrigger arguments and classpath.

async run()[source]

Run the trigger.

If _polling_interval > 0, this pools Livy for batch termination asynchronously. Otherwise the success response is created immediately.

async poll_for_termination(batch_id)[source]

Pool Livy for batch termination asynchronously.

Parameters

batch_id (int | str) – id of the batch session to monitor.

Was this entry helpful?