airflow.contrib.hooks.ftp_hook

Module Contents

airflow.contrib.hooks.ftp_hook.mlsd(conn, path='', facts=None)[source]
BACKPORT FROM PYTHON3 FTPLIB.

List a directory in a standardized format by using MLSD command (RFC-3659). If path is omitted the current directory is assumed. “facts” is a list of strings representing the type of information desired (e.g. [“type”, “size”, “perm”]).

Return a generator object yielding a tuple of two elements for every file found in path. First element is the file name, the second one is a dictionary including a variable number of “facts” depending on the server and whether “facts” argument has been provided.

class airflow.contrib.hooks.ftp_hook.FTPHook(ftp_conn_id='ftp_default')[source]

Bases: airflow.hooks.base_hook.BaseHook

Interact with FTP.

Errors that may occur throughout but should be handled downstream. You can specify mode for data transfers in the extra field of your connection as {"passive": "true"}.

__enter__(self)[source]
__exit__(self, exc_type, exc_val, exc_tb)[source]
get_conn(self)[source]

Returns a FTP connection object

close_conn(self)[source]

Closes the connection. An error will occur if the connection wasn’t ever opened.

describe_directory(self, path)[source]

Returns a dictionary of {filename: {attributes}} for all files on the remote system (where the MLSD command is supported).

Parameters

path (str) – full path to the remote directory

list_directory(self, path, nlst=False)[source]

Returns a list of files on the remote system.

Parameters

path (str) – full path to the remote directory to list

create_directory(self, path)[source]

Creates a directory on the remote system.

Parameters

path (str) – full path to the remote directory to create

delete_directory(self, path)[source]

Deletes a directory on the remote system.

Parameters

path (str) – full path to the remote directory to delete

retrieve_file(self, remote_full_path, local_full_path_or_buffer, callback=None)[source]

Transfers the remote file to a local location.

If local_full_path_or_buffer is a string path, the file will be put at that location; if it is a file-like buffer, the file will be written to the buffer but not closed.

Parameters

  • remote_full_path (str) – full path to the remote file

  • local_full_path_or_buffer (str or file-like buffer) – full path to the local file or a file-like buffer

  • callback (callable) – callback which is called each time a block of data is read. if you do not use a callback, these blocks will be written to the file or buffer passed in. if you do pass in a callback, note that writing to a file or buffer will need to be handled inside the callback. [default: output_handle.write()]

hook = FTPHook(ftp_conn_id='my_conn')

remote_path = '/path/to/remote/file'
local_path = '/path/to/local/file'

# with a custom callback (in this case displaying progress on each read)
def print_progress(percent_progress):
    self.log.info('Percent Downloaded: %s%%' % percent_progress)

total_downloaded = 0
total_file_size = hook.get_size(remote_path)
output_handle = open(local_path, 'wb')
def write_to_file_with_progress(data):
    total_downloaded += len(data)
    output_handle.write(data)
    percent_progress = (total_downloaded / total_file_size) * 100
    print_progress(percent_progress)
hook.retrieve_file(remote_path, None, callback=write_to_file_with_progress)

# without a custom callback data is written to the local_path
hook.retrieve_file(remote_path, local_path)
store_file(self, remote_full_path, local_full_path_or_buffer)[source]

Transfers a local file to the remote location.

If local_full_path_or_buffer is a string path, the file will be read from that location; if it is a file-like buffer, the file will be read from the buffer but not closed.

Parameters

  • remote_full_path (str) – full path to the remote file

  • local_full_path_or_buffer (str or file-like buffer) – full path to the local file or a file-like buffer

delete_file(self, path)[source]

Removes a file on the FTP Server.

Parameters

path (str) – full path to the remote file

rename(self, from_name, to_name)[source]

Rename a file.

Parameters

  • from_name – rename file from name

  • to_name – rename file to name

get_mod_time(self, path)[source]

Returns a datetime object representing the last time the file was modified

Parameters

path (string) – remote file path

get_size(self, path)[source]

Returns the size of a file (in bytes)

Parameters

path (string) – remote file path

class airflow.contrib.hooks.ftp_hook.FTPSHook[source]

Bases: airflow.contrib.hooks.ftp_hook.FTPHook

get_conn(self)[source]

Returns a FTPS connection object.

Was this entry helpful?