airflow.providers.plexus.operators.job

Module Contents

Classes

PlexusJobOperator

Submits a Plexus job.

Attributes

logger

airflow.providers.plexus.operators.job.logger[source]
class airflow.providers.plexus.operators.job.PlexusJobOperator(job_params, **kwargs)[source]

Bases: airflow.models.BaseOperator

Submits a Plexus job.

Parameters

job_params (Dict) -- parameters required to launch a job.

Required job parameters are the following

  • "name": job name created by user.

  • "app": name of the application to run. found in Plexus UI.

  • "queue": public cluster name. found in Plexus UI.

  • "num_nodes": number of nodes.

  • "num_cores": number of cores per node.

execute(self, context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

construct_job_params(self, hook)[source]

Creates job_params dict for api call to launch a Plexus job.

Some parameters required to launch a job are not available to the user in the Plexus UI. For example, an app id is required, but only the app name is provided in the UI. This function acts as a backend lookup of the required param value using the user-provided value.

Parameters

hook (Any) -- plexus hook object

Was this entry helpful?