airflow.providers.sftp.operators.sftp

This module contains SFTP operator.

Module Contents

Classes

SFTPOperation

Operation that can be used with SFTP.

SFTPOperator

SFTPOperator for transferring files from remote host to local or vice a versa.

class airflow.providers.sftp.operators.sftp.SFTPOperation[source]

Operation that can be used with SFTP.

PUT = 'put'[source]
GET = 'get'[source]
class airflow.providers.sftp.operators.sftp.SFTPOperator(*, ssh_hook=None, sftp_hook=None, ssh_conn_id=None, remote_host=None, local_filepath, remote_filepath, operation=SFTPOperation.PUT, confirm=True, create_intermediate_dirs=False, **kwargs)[source]

Bases: airflow.models.BaseOperator

SFTPOperator for transferring files from remote host to local or vice a versa.

This operator uses sftp_hook to open sftp transport channel that serve as basis for file transfer.

Parameters
  • ssh_conn_id (str | None) – ssh connection id from airflow Connections. ssh_conn_id will be ignored if ssh_hook or sftp_hook is provided.

  • sftp_hook (airflow.providers.sftp.hooks.sftp.SFTPHook | None) – predefined SFTPHook to use Either sftp_hook or ssh_conn_id needs to be provided.

  • ssh_hook (airflow.providers.ssh.hooks.ssh.SSHHook | None) – Deprecated - predefined SSHHook to use for remote execution Use sftp_hook instead.

  • remote_host (str | None) – remote host to connect (templated) Nullable. If provided, it will replace the remote_host which was defined in sftp_hook/ssh_hook or predefined in the connection of ssh_conn_id.

  • local_filepath (str | list[str]) – local file path or list of local file paths to get or put. (templated)

  • remote_filepath (str | list[str]) – remote file path or list of remote file paths to get or put. (templated)

  • operation (str) – specify operation ‘get’ or ‘put’, defaults to put

  • confirm (bool) – specify if the SFTP operation should be confirmed, defaults to True

  • create_intermediate_dirs (bool) –

    create missing intermediate directories when copying from remote to local and vice-versa. Default is False.

    Example: The following task would copy file.txt to the remote host at /tmp/tmp1/tmp2/ while creating tmp,``tmp1`` and tmp2 if they don’t exist. If the parameter is not passed it would error as the directory does not exist.

    put_file = SFTPOperator(
        task_id="test_sftp",
        ssh_conn_id="ssh_default",
        local_filepath="/tmp/file.txt",
        remote_filepath="/tmp/tmp1/tmp2/file.txt",
        operation="put",
        create_intermediate_dirs=True,
        dag=dag,
    )
    

template_fields: Sequence[str] = ('local_filepath', 'remote_filepath', 'remote_host')[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_openlineage_facets_on_start()[source]

Return OpenLineage datasets.

Dataset will have the following structure:

input: file://<local_host>/path output: file://<remote_host>:<remote_port>/path.

Was this entry helpful?