• Home
• airflow.providers.sftp
• airflow.providers.sftp.hooks
• airflow.providers.sftp.hooks.sftp

↑↓ Navigate ⏎ Select Esc Close

airflow.providers.sftp.hooks.sftp

This module contains SFTP hook.

Attributes

CHUNK_SIZE

Classes

SFTPOperation	SFTP operation constants.
SFTPHook	Interact with SFTP.
SFTPHookAsync	Interact with an SFTP server via asyncssh package.

Functions

handle_connection_management(func)

Run the wrapped hook method inside the hook's managed connection.

Module Contents

airflow.providers.sftp.hooks.sftp.CHUNK_SIZE = 65536

class airflow.providers.sftp.hooks.sftp.SFTPOperation

Bases: str, enum.Enum

SFTP operation constants.

GET = 'get'

PUT = 'put'

DELETE = 'delete'

airflow.providers.sftp.hooks.sftp.handle_connection_management(func)

Run the wrapped hook method inside the hook’s managed connection.

Both SFTPHook and SFTPHookAsync expose get_managed_conn(), which opens the connection on first entry and reuses it for nested entries, so a decorated method calling other decorated methods shares one connection with them.

class airflow.providers.sftp.hooks.sftp.SFTPHook(ssh_conn_id='sftp_default', host_proxy_cmd=None, use_managed_conn=True, *args, **kwargs)

Bases: airflow.providers.ssh.hooks.ssh.SSHHook

Interact with SFTP.

This hook inherits the SSH hook. Please refer to SSH hook for the input arguments.

Pitfalls::

• In contrast with FTPHook describe_directory only returns size, type and modify. It doesn’t return unix.owner, unix.mode, perm, unix.group and unique.

• If no mode is passed to create_directory it will be created with 777 permissions.

Errors that may occur throughout but should be handled downstream.

For consistency reasons with SSHHook, the preferred parameter is “ssh_conn_id”.

Parameters:

ssh_conn_id (str | None) – The sftp connection id

conn_name_attr = 'ssh_conn_id'

default_conn_name = 'sftp_default'

conn_type = 'sftp'

hook_name = 'SFTP'

classmethod get_ui_field_behaviour()

Return custom UI field behaviour for SSH connection.

conn: paramiko.sftp_client.SFTPClient | None = None

use_managed_conn = True

ssh_conn_id = 'sftp_default'

get_conn()

Open an SFTP connection to the remote host.

close_conn()

Close the SFTP connection.

get_managed_conn()

Context manager that closes the connection after use.

get_conn_count()

Get the number of open connections.

describe_directory(path)

Get file information in a directory on the remote system.

The return format is {filename: {attributes}}. The remote system support the MLSD command.

Parameters:

path (str) – full path to the remote directory

list_directory(path, recursive=False)

List files in a directory on the remote system.

Lists one-level entry names under the given directory path.

If recursive=True, returns files recursively as paths relative to path.

Parameters:

• path (str) – full path to the remote directory to list

• recursive (bool) – Whether to recursively list descendants.

Returns:

List of entry names found under the directory, or None if the directory does not exist.

Return type:

list[str] | None

list_directory_with_attr(path)

List files in a directory on the remote system including their SFTPAttributes.

Parameters:

path (str) – full path to the remote directory to list

mkdir(path, mode=511)

Create a directory on the remote system.

The default mode is 0o777, but on some systems, the current umask value may be first masked out.

Parameters:

• path (str) – full path to the remote directory to create

• mode (int) – int permissions of octal mode for directory

isdir(path)

Check if the path provided is a directory.

Parameters:

path (str) – full path to the remote directory to check

isfile(path)

Check if the path provided is a file.

Parameters:

path (str) – full path to the remote file to check

create_directory(path, mode=511)

Create a directory on the remote system.

The default mode is 0o777, but on some systems, the current umask value may be first masked out. Different from mkdir(), this function attempts to create parent directories if needed, and returns silently if the target directory already exists.

Parameters:

• path (str) – full path to the remote directory to create

• mode (int) – int permissions of octal mode for directory

delete_directory(path, include_files=False)

Delete a directory on the remote system.

Parameters:

path (str) – full path to the remote directory to delete

retrieve_file(remote_full_path, local_full_path, prefetch=True)

Transfer the remote file to a local location.

If local_full_path is a string path, the file will be put at that location.

Parameters:

• remote_full_path (str) – full path to the remote file

• local_full_path (str) – full path to the local file or a file-like buffer

• prefetch (bool) – controls whether prefetch is performed (default: True)

store_file(remote_full_path, local_full_path, confirm=True)

Transfer a local file to the remote location.

If local_full_path_or_buffer is a string path, the file will be read from that location.

Parameters:

• remote_full_path (str) – full path to the remote file

• local_full_path (str) – full path to the local file or a file-like buffer

delete_file(path)

Remove a file on the server.

Parameters:

path (str) – full path to the remote file

retrieve_directory(remote_full_path, local_full_path, prefetch=True)

Transfer the remote directory to a local location.

If local_full_path is a string path, the directory will be put at that location.

Parameters:

• remote_full_path (str) – full path to the remote directory

• local_full_path (str) – full path to the local directory

• prefetch (bool) – controls whether prefetch is performed (default: True)

retrieve_directory_concurrently(remote_full_path, local_full_path, workers=os.cpu_count() or 2, prefetch=True)

Transfer the remote directory to a local location concurrently.

If local_full_path is a string path, the directory will be put at that location.

Parameters:

• remote_full_path (str) – full path to the remote directory

• local_full_path (str) – full path to the local directory

• prefetch (bool) – controls whether prefetch is performed (default: True)

• workers (int) – number of workers to use for concurrent transfer (default: number of CPUs or 2 if undetermined)

store_directory(remote_full_path, local_full_path, confirm=True)

Transfer a local directory to the remote location.

If local_full_path is a string path, the directory will be read from that location.

Parameters:

• remote_full_path (str) – full path to the remote directory

• local_full_path (str) – full path to the local directory

store_directory_concurrently(remote_full_path, local_full_path, confirm=True, workers=os.cpu_count() or 2)

Transfer a local directory to the remote location concurrently.

If local_full_path is a string path, the directory will be read from that location.

Parameters:

• remote_full_path (str) – full path to the remote directory

• local_full_path (str) – full path to the local directory

• confirm (bool) – whether to confirm the file size after transfer (default: True)

• workers (int) – number of workers to use for concurrent transfer (default: number of CPUs or 2 if undetermined)

get_mod_time(path)

Get an entry’s modification time.

Parameters:

path (str) – full path to the remote file

path_exists(path)

Whether a remote entity exists.

Parameters:

path (str) – full path to the remote file or directory

walktree(path, fcallback, dcallback, ucallback, recurse=True)

Recursively descend, depth first, the directory tree at path.

This calls discrete callback functions for each regular file, directory, and unknown file type.

Parameters:

• path (str) – root of remote directory to descend, use ‘.’ to start at pwd

• fcallback (callable) – callback function to invoke for a regular file. (form: func(str))

• dcallback (callable) – callback function to invoke for a directory. (form: func(str))

• ucallback (callable) – callback function to invoke for an unknown file type. (form: func(str))

• recurse (bool) – Default: True - should it recurse

get_tree_map(path, prefix=None, delimiter=None)

Get tuple with recursive lists of files, directories and unknown paths.

It is possible to filter results by giving prefix and/or delimiter parameters.

Parameters:

• path (str) – path from which tree will be built

• prefix (str | None) – if set paths will be added if start with prefix

• delimiter (str | None) – if set paths will be added if end with delimiter

Returns:

tuple with list of files, dirs and unknown items

Return type:

tuple[list[str], list[str], list[str]]

test_connection()

Test the SFTP connection by calling path with directory.

get_file_by_pattern(path, fnmatch_pattern)

Get the first matching file based on the given fnmatch type pattern.

Parameters:

• path – path to be checked

• fnmatch_pattern – The pattern that will be matched with fnmatch

Returns:

string containing the first found file, or an empty string if none matched

Return type:

str

get_files_by_pattern(path, fnmatch_pattern)

Get all matching files based on the given fnmatch type pattern.

Parameters:

• path – path to be checked

• fnmatch_pattern – The pattern that will be matched with fnmatch

Returns:

list of string containing the found files, or an empty list if none matched

Return type:

list[str]

transfer(operation, local_filepath, remote_filepath, confirm=True, create_intermediate_dirs=False, concurrency=1, prefetch=True)

Perform a synchronous SFTP transfer operation (GET, PUT, or DELETE).

Centralizes transfer logic so both the operator and the trigger can delegate to the hook, in line with the DRY principle.

Parameters:

• operation (str) – The SFTP operation - put, get, or delete.

• local_filepath (str | list[str] | None) – Local file path(s).

• remote_filepath (str | list[str]) – Remote file path(s).

• confirm (bool) – Whether to confirm file size after PUT (default: True).

• create_intermediate_dirs (bool) – Create missing intermediate directories (default: False).

• concurrency (int) – Number of threads for directory transfers (default: 1).

• prefetch (bool) – Whether to prefetch during GET (default: True).

class airflow.providers.sftp.hooks.sftp.SFTPHookAsync(sftp_conn_id=default_conn_name, host=None, port=None, username=None, password=None, known_hosts=default_known_hosts, key_file='', passphrase='', private_key='')

Bases: airflow.providers.common.compat.sdk.BaseHook

Interact with an SFTP server via asyncssh package.

Parameters:

• sftp_conn_id (str) – SFTP connection ID to be used for connecting to SFTP server

• host (str | None) – hostname of the SFTP server

• port (int | None) – port of the SFTP server

• username (str | None) – username used when authenticating to the SFTP server

• password (str | None) – password used when authenticating to the SFTP server. Can be left blank if using a key file

• known_hosts (str) – path to the known_hosts file on the local file system. Defaults to ~/.ssh/known_hosts.

• key_file (str) – path to the client key file used for authentication to SFTP server

• passphrase (str) – passphrase used with the key_file for authentication to SFTP server

conn_name_attr = 'ssh_conn_id'

default_conn_name = 'sftp_default'

conn_type = 'sftp'

hook_name = 'SFTP'

default_known_hosts = '~/.ssh/known_hosts'

sftp_conn_id = 'sftp_default'

host = None

port = None

username = None

password = None

known_hosts: bytes | str

key_file = ''

passphrase = ''

private_key = ''

conn: asyncssh.SFTPClient | None = None

async get_managed_conn()

Context manager sharing one SSH connection and SFTP client across nested uses.

The connection is opened on the first entry, reused by any entry made while it is still open, and closed when the last user exits, mirroring SFTPHook.get_managed_conn(). Hook methods are wrapped in it, so wrapping several calls in this context manager makes them run over a single connection.

get_conn_count()

Get the number of users currently sharing the open connection.

async retrieve_file(remote_full_path, local_full_path, chunk_size=CHUNK_SIZE, prefetch=True)

Transfer the remote file to a local location asynchronously.

If local_full_path is a string or PathLike path, the file will be put at that location. If it is a BytesIO or other binary file-like object, the file will be streamed into it.

Parameters:

• remote_full_path (str) – Full path to the remote file.

• local_full_path (str | os.PathLike[str] | IO[bytes]) – Full path to the local file or a binary file-like buffer.

• chunk_size (int) – Size of chunks to read at a time (default: 64KB).

• prefetch (bool) – Whether to allow read-ahead requests to be sent concurrently (default: True). When False, only one request is kept in flight at a time, mirroring SFTPHook.retrieve_file()’s prefetch semantics.

async store_file(remote_full_path, local_full_path, confirm=True)

Transfer a local file to the remote location.

If local_full_path is a path, the file will be read from that location. If it is a binary file-like object, the content will be uploaded from the stream. Raw bytes values are not accepted directly; wrap bytes in BytesIO.

Parent directories for remote_full_path are created when missing.

Parameters:

• remote_full_path (str) – full path to the remote file

• local_full_path (str | os.PathLike[str] | IO[bytes]) – full path to the local file or a binary file-like buffer

• confirm (bool) – whether to verify the remote file size matches the local size after upload (default: True), mirroring SFTPHook.store_file()’s confirm semantics.

async mkdir(path)

Create a directory on the remote system asynchronously.

The default permissions are determined by the server. Parent directories are created as needed.

Parameters:

path (str) – Full path to the remote directory to create.

async list_directory(path='', recursive=False)

List files in a directory on the remote system asynchronously.

Lists one-level entry names under the given directory path.

If recursive=True, returns files recursively as paths relative to path.

Parameters:

• path (str) – Full path to the remote directory to list.

• recursive (bool) – Whether to recursively list descendants.

Returns:

List of entry names found under the directory, or None if the directory does not exist.

Return type:

list[str] | None

async walktree(path, fcallback, dcallback, ucallback, recurse=True)

Recursively descend, depth first, the directory tree at path.

This mirrors SFTPHook.walktree() contract and calls callback functions for regular files, directories, and unknown file types.

async read_directory(path='')

Return a list of files along with their attributes on the SFTP server at the provided path.

async get_files_and_attrs_by_pattern(path='', fnmatch_pattern='')

Get the files along with their attributes matching the pattern (e.g. *.pdf) at the provided path.

if one exists. Otherwise, raises an AirflowException to be handled upstream for deferring

async get_mod_time(path)

Make SFTP async connection.

Looks for last modified time in the specific file path and returns last modification time for

the file path.

Parameters:

path (str) – full path to the remote file

async sense_files_by_pattern(path, fnmatch_pattern, newer_than=None)

Return the names of files at path matching fnmatch_pattern.

If newer_than is provided, only files modified after that timestamp are returned; files without a reported modification time are skipped in that case.

Parameters:

• path (str) – directory on the SFTP server to search for files matching the pattern

• fnmatch_pattern (str) – pattern used to match filenames, see the fnmatch std library module

• newer_than (datetime.datetime | None) – if provided, only files modified after this UTC timestamp are returned

async sense_path(path, newer_than=None)

Return whether path exists and, if newer_than is provided, was modified since.

Parameters:

• path (str) – full path to the remote file

• newer_than (datetime.datetime | None) – if provided, the file must have been modified after this UTC timestamp

async isdir(path)

Check if the path provided is a directory.

Parameters:

path (str) – full path to the remote directory to check

async path_exists(path)

Whether a remote entity exists.

Parameters:

path (str) – full path to the remote file or directory

async create_directory(path)

Create a directory (and any missing parents) on the remote system asynchronously.

Returns silently if the target directory already exists, mirroring SFTPHook.create_directory().

Parameters:

path (str) – full path to the remote directory to create

async delete_file(path)

Remove a file on the server asynchronously.

Parameters:

path (str) – full path to the remote file

async delete_directory(path, include_files=False)

Delete a directory on the remote system asynchronously.

Parameters:

• path (str) – full path to the remote directory to delete

• include_files (bool) – whether to recursively delete the directory’s contents first

async get_tree_map(path, prefix=None, delimiter=None)

Get tuple with recursive lists of files, directories and unknown paths asynchronously.

It is possible to filter results by giving prefix and/or delimiter parameters.

Parameters:

• path (str) – path from which tree will be built

• prefix (str | None) – if set paths will be added if start with prefix

• delimiter (str | None) – if set paths will be added if end with delimiter

Returns:

tuple with list of files, dirs and unknown items

Return type:

tuple[list[str], list[str], list[str]]

async retrieve_directory(remote_full_path, local_full_path, prefetch=True)

Transfer the remote directory to a local location asynchronously.

The whole tree is walked and downloaded over a single connection.

Parameters:

• remote_full_path (str) – full path to the remote directory

• local_full_path (str) – full path to the local directory

• prefetch (bool) – whether read-ahead requests are sent concurrently (default: True)

async store_directory(remote_full_path, local_full_path, confirm=True)

Transfer a local directory to the remote location asynchronously.

The whole tree is created and uploaded over a single connection.

Parameters:

• remote_full_path (str) – full path to the remote directory

• local_full_path (str) – full path to the local directory

• confirm (bool) – whether to verify each uploaded file’s size (default: True)

async transfer(operation, local_filepath, remote_filepath, confirm=True, create_intermediate_dirs=False, concurrency=1, prefetch=True)

Perform an SFTP transfer operation (GET, PUT, or DELETE) using native async I/O.

Mirrors SFTPHook.transfer(), including directory transfers, missing-file deletes, and the confirm/prefetch options, so deferrable=True behaves the same as the synchronous path. The whole call runs over a single connection and concurrency bounds how many top-level paths are in flight on it at once; files inside a directory are transferred sequentially. The synchronous path differs there: it opens one connection per worker and transfers directory contents concurrently.

Previous Next

Was this entry helpful?