airflow.providers.microsoft.azure.transfers.sftp_to_wasb

This module contains SFTP to Azure Blob Storage operator.

Module Contents

Classes

SFTPToWasbOperator

Transfer files to Azure Blob Storage from SFTP server.

Attributes

WILDCARD

SftpFile

airflow.providers.microsoft.azure.transfers.sftp_to_wasb.WILDCARD = '*'[source]
airflow.providers.microsoft.azure.transfers.sftp_to_wasb.SftpFile[source]
class airflow.providers.microsoft.azure.transfers.sftp_to_wasb.SFTPToWasbOperator(*, sftp_source_path, container_name, blob_prefix='', sftp_conn_id='sftp_default', wasb_conn_id='wasb_default', load_options=None, move_object=False, wasb_overwrite_object=False, create_container=False, **kwargs)[source]

Bases: airflow.models.BaseOperator

Transfer files to Azure Blob Storage from SFTP server.

See also

For more information on how to use this operator, take a look at the guide: Transfer Data from SFTP Source Path to Blob Storage

Parameters

  • sftp_source_path (str) – The sftp remote path. This is the specified file path for downloading the single file or multiple files from the SFTP server. You can use only one wildcard within your path. The wildcard can appear inside the path or at the end of the path.

  • container_name (str) – Name of the container.

  • blob_prefix (str) – Prefix to name a blob.

  • sftp_conn_id (str) – The sftp connection id. The name or identifier for establishing a connection to the SFTP server.

  • wasb_conn_id (str) – Reference to the wasb connection.

  • load_options (dict | None) – Optional keyword arguments that WasbHook.load_file() takes.

  • move_object (bool) – When move object is True, the object is moved instead of copied to the new location. This is the equivalent of a mv command as opposed to a cp command.

  • wasb_overwrite_object (bool) – Whether the blob to be uploaded should overwrite the current data. When wasb_overwrite_object is True, it will overwrite the existing data. If set to False, the operation might fail with ResourceExistsError in case a blob object already exists.

  • create_container (bool) – Attempt to create the target container prior to uploading the blob. This is useful if the target container may not exist yet. Defaults to False.

property source_path_contains_wildcard: bool[source]

Checks if the SFTP source path contains a wildcard.

template_fields: Sequence[str] = ('sftp_source_path', 'container_name', 'blob_prefix')[source]
dry_run()[source]

Performs dry run for the operator - just render template fields.

execute(context)[source]

Upload a file from SFTP to Azure Blob Storage.

get_sftp_files_map()[source]

Get SFTP files from the source path, it may use a WILDCARD to this end.

get_tree_behavior()[source]

Extracts from source path the tree behavior to interact with the remote folder

check_wildcards_limit()[source]

Check if there are multiple wildcards used in the SFTP source path.

sftp_hook()[source]

Property of sftp hook to be re-used.

get_full_path_blob(file)[source]

Get a blob name based on the previous name and a blob_prefix variable

copy_files_to_wasb(sftp_files)[source]

Upload a list of files from sftp_files to Azure Blob Storage with a new Blob Name.

delete_files(uploaded_files)[source]

Delete files at SFTP which have been moved to Azure Blob Storage.

Was this entry helpful?