airflow.providers.apache.livy.operators.livy¶
Classes¶
Airflow <3.3 stub, task_state_store unavailable, always submits fresh. |
|
Wraps the Apache Livy batch REST API, allowing to submit a Spark application to the underlying cluster. |
Module Contents¶
- class airflow.providers.apache.livy.operators.livy.ResumableJobMixin(*, durable=_DURABLE_UNSET, **kwargs)[source]¶
Airflow <3.3 stub, task_state_store unavailable, always submits fresh.
- class airflow.providers.apache.livy.operators.livy.LivyOperator(*, file, class_name=None, args=None, conf=None, jars=None, py_files=None, files=None, driver_memory=None, driver_cores=None, executor_memory=None, executor_cores=None, num_executors=None, archives=None, queue=None, name=None, proxy_user=None, livy_conn_id='livy_default', livy_conn_auth_type=None, livy_endpoint_prefix=None, polling_interval=0, extra_options=None, extra_headers=None, retry_args=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), openlineage_inject_parent_job_info=conf.getboolean('openlineage', 'spark_inject_parent_job_info', fallback=False), openlineage_inject_transport_info=conf.getboolean('openlineage', 'spark_inject_transport_info', fallback=False), durable=None, **kwargs)[source]¶
Bases:
airflow.sdk.ResumableJobMixin,airflow.providers.common.compat.sdk.BaseOperatorWraps the Apache Livy batch REST API, allowing to submit a Spark application to the underlying cluster.
- Parameters:
file (str) – path of the file containing the application to execute (required). (templated)
class_name (str | None) – name of the application Java/Spark main class. (templated)
args (collections.abc.Sequence[str | int | float] | None) – application command line arguments. (templated)
jars (collections.abc.Sequence[str] | None) – jars to be used in this sessions. (templated)
py_files (collections.abc.Sequence[str] | None) – python files to be used in this session. (templated)
files (collections.abc.Sequence[str] | None) – files to be used in this session. (templated)
driver_memory (str | None) – amount of memory to use for the driver process. (templated)
driver_cores (int | str | None) – number of cores to use for the driver process. (templated)
executor_memory (str | None) – amount of memory to use per executor process. (templated)
executor_cores (int | str | None) – number of cores to use for each executor. (templated)
num_executors (int | str | None) – number of executors to launch for this session. (templated)
archives (collections.abc.Sequence[str] | None) – archives to be used in this session. (templated)
queue (str | None) – name of the YARN queue to which the application is submitted. (templated)
name (str | None) – name of this session. (templated)
conf (dict[Any, Any] | None) – Spark configuration properties. (templated)
proxy_user (str | None) – user to impersonate when running the job. (templated)
livy_conn_id (str) – reference to a pre-defined Livy Connection.
livy_conn_auth_type (Any | None) – The auth type for the Livy Connection.
polling_interval (int) – time in seconds between polling for job completion. Don’t poll for values <= 0
extra_options (dict[str, Any] | None) – A dictionary of options, where key is string and value depends on the option that’s being modified.
extra_headers (dict[str, Any] | None) – A dictionary of headers passed to the HTTP request to livy.
retry_args (dict[str, Any] | None) – Arguments which define the retry behaviour. See Tenacity documentation at https://github.com/jd/tenacity
deferrable (bool) – Run operator in the deferrable mode
durable (bool | None) – When True (the default) and the operator waits synchronously (
deferrable=Falsewithpolling_interval > 0), the Livy batch id is persisted before polling so a worker crash reconnects to the running batch on retry instead of submitting a duplicate. Requires Airflow 3.3+ (task_state_store); a no-op on earlier versions.
- template_fields: collections.abc.Sequence[str] = ('spark_params',)[source]¶
- property hook: airflow.providers.apache.livy.hooks.livy.LivyHook[source]¶
Get valid hook.
- Returns:
LivyHook
- Return type:
- execute(context)[source]¶
Derive when creating an operator.
The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
- submit_job(context)[source]¶
Submit the job to the external system. Return its external ID.
The returned ID must not be
None, aNonereturn is treated as “no ID available” and the ID will not be persisted to task state.
- get_job_status(external_id, context)[source]¶
Query the external system for the current job status.
contextis provided so implementations can use it if needed to implement advanced features such as:cache terminal status to
task_state_storewhen the remote resource may be ephemeral (e.g. a K8s driver pod that gets garbage-collected after completion),
- is_job_active(status)[source]¶
Return True if the job is still running and can be reconnected to.
statusis a raw string returned by the external system — not an Airflow enum. Its values are backend-specific (e.g."RUNNING","Pending","ContainerCreating").
- is_job_succeeded(status)[source]¶
Return True if the job completed successfully.
statusis a raw string returned by the external system — not an Airflow enum. Its values are backend-specific (e.g."FINISHED","Succeeded").
- poll_until_complete(external_id, context)[source]¶
Block until the job reaches a terminal state. Raise on failure.
- get_job_result(external_id, context)[source]¶
Return the job result after completion. Return None if not applicable.