airflow.providers.ssh.operators.ssh_remote_job

SSH Remote Job Operator for deferrable remote command execution.

Classes

SSHRemoteJobOperator

Execute a command on a remote host via SSH with deferrable monitoring.

Module Contents

class airflow.providers.ssh.operators.ssh_remote_job.SSHRemoteJobOperator(*, ssh_conn_id, command, remote_host=None, environment=None, remote_base_dir=None, poll_interval=5, log_chunk_size=65536, timeout=None, cleanup='never', remote_os='auto', skip_on_exit_code=None, conn_timeout=None, banner_timeout=30.0, conn_retry_attempts=5, cleanup_retries=3, command_timeout=30.0, max_reconnect_attempts=5, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Execute a command on a remote host via SSH with deferrable monitoring.

This operator submits a job to run detached on the remote host, then uses a trigger to asynchronously monitor the job status and stream logs. This approach is resilient to network interruptions as the remote job continues running independently of the SSH connection.

The remote job is wrapped to: - Run detached from the SSH session (via nohup on POSIX, Start-Process on Windows) - Redirect stdout/stderr to a log file - Write the exit code to a file on completion

Parameters:

  • ssh_conn_id (str) – SSH connection ID from Airflow Connections

  • command (str) – Command to execute on the remote host (templated)

  • remote_host (str | None) – Override the host from the connection (templated)

  • environment (dict[str, str] | None) – Environment variables to set for the command (templated)

  • remote_base_dir (str | None) – Base directory for job artifacts (templated). Defaults to /tmp/airflow-ssh-jobs on POSIX, C:\Windows\Temp\airflow-ssh-jobs on Windows

  • poll_interval (int) – Seconds between status polls (default: 5)

  • log_chunk_size (int) – Max bytes to read per poll (default: 65536)

  • timeout (int | None) – Hard timeout in seconds for the entire operation

  • cleanup (Literal['never', 'on_success', 'always']) – When to clean up remote job directory: ‘never’, ‘on_success’, or ‘always’ (default: ‘never’)

  • remote_os (Literal['auto', 'posix', 'windows']) – Remote operating system: ‘auto’, ‘posix’, or ‘windows’ (default: ‘auto’)

  • skip_on_exit_code (int | collections.abc.Container[int] | None) – Exit codes that should skip the task instead of failing

  • conn_timeout (int | None) – SSH connection timeout in seconds

  • banner_timeout (float) – Timeout waiting for SSH banner in seconds

  • conn_retry_attempts (int) – How many times to attempt the initial SSH connection for submission/cleanup before failing (default 5). Helps when many mapped tasks hit the same host at once and sshd transiently refuses connections (MaxStartups).

  • cleanup_retries (int) – How many times to attempt remote directory cleanup before giving up and leaving the directory in place (default 3). Prevents a transient SSH failure during cleanup from orphaning the job directory on the remote host.

  • command_timeout (float) – Per-command timeout in seconds for the trigger’s status/log polls (default 30.0).

  • max_reconnect_attempts (int) – Consecutive connection failures the trigger tolerates (with backoff) before failing the task while monitoring the remote job (default 5).

Note

A large expand() fan-out opens many SSH connections against one host. The remote sshd throttles concurrent unauthenticated connections via MaxStartups (default 10:30:100); when exceeded it drops connections, surfacing as paramiko ... Error reading SSH protocol banner. For high fan-out, raise MaxStartups on the server. The directory /tmp/airflow-ssh-jobs (POSIX) is only cleaned when cleanup is set and the job reaches completion, so also consider a server-side TTL reaper (for example systemd-tmpfiles) for jobs that are killed or time out.

template_fields: collections.abc.Sequence[str] = ('command', 'environment', 'remote_host', 'remote_base_dir')[source]
template_ext: collections.abc.Sequence[str] = ('.sh', '.bash', '.ps1')[source]
template_fields_renderers[source]
ui_color = '#e4f0e8'[source]
ssh_conn_id[source]
command[source]
remote_host = None[source]
environment = None[source]
remote_base_dir = None[source]
poll_interval = 5[source]
log_chunk_size = 65536[source]
timeout = None[source]
cleanup = 'never'[source]
remote_os = 'auto'[source]
conn_timeout = None[source]
banner_timeout = 30.0[source]
conn_retry_attempts = 5[source]
cleanup_retries[source]
command_timeout = 30.0[source]
max_reconnect_attempts = 5[source]
skip_on_exit_code[source]
property ssh_hook: airflow.providers.ssh.hooks.ssh.SSHHook[source]

Create the SSH hook for command submission.

execute(context)[source]

Submit the remote job and defer to the trigger for monitoring.

Parameters:

context (airflow.providers.common.compat.sdk.Context) – Airflow task context

execute_complete(context, event)[source]

Handle trigger events and re-defer if job is still running.

Parameters:
on_kill()[source]

Attempt to kill the remote process when the task is killed.

Since the operator is recreated after deferral, instance variables may not be set. We retrieve job information from XCom if needed.

Was this entry helpful?