airflow.providers.plexus.operators.job

Module Contents

airflow.providers.plexus.operators.job.logger[source]
class airflow.providers.plexus.operators.job.PlexusJobOperator(job_params: Dict, **kwargs)[source]

Bases: airflow.models.BaseOperator

Submits a Plexus job.

Parameters

job_params (dict) -- parameters required to launch a job.

Required job parameters are the following

  • "name": job name created by user.

  • "app": name of the application to run. found in Plexus UI.

  • "queue": public cluster name. found in Plexus UI.

  • "num_nodes": number of nodes.

  • "num_cores": number of cores per node.

execute(self, context: Any)[source]
_api_lookup(self, param: str, hook)[source]
construct_job_params(self, hook: Any)[source]

Creates job_params dict for api call to launch a Plexus job.

Some parameters required to launch a job are not available to the user in the Plexus UI. For example, an app id is required, but only the app name is provided in the UI. This function acts as a backend lookup of the required param value using the user-provided value.

Parameters

hook (airflow hook) -- plexus hook object

Was this entry helpful?