airflow.providers.microsoft.azure.transfers.local_to_adls¶
Module Contents¶
Classes¶
Upload file(s) to Azure Data Lake. |
- class airflow.providers.microsoft.azure.transfers.local_to_adls.LocalFilesystemToADLSOperator(*, local_path, remote_path, overwrite=True, nthreads=64, buffersize=4194304, blocksize=4194304, extra_upload_options=None, azure_data_lake_conn_id='azure_data_lake_default', **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorUpload file(s) to Azure Data Lake.
See also
For more information on how to use this operator, take a look at the guide: LocalFilesystemToADLSOperator
- Parameters
local_path (str) – local path. Can be single file, directory (in which case, upload recursively) or glob pattern. Recursive glob patterns using ** are not supported
remote_path (str) – Remote path to upload to; if multiple files, this is the directory root to write within
nthreads (int) – Number of threads to use. If None, uses the number of cores.
overwrite (bool) – Whether to forcibly overwrite existing files/directories. If False and remote path is a directory, will quit regardless if any files would be overwritten or not. If True, only matching filenames are actually overwritten
buffersize (int) – int [2**22] Number of bytes for internal buffer. This block cannot be bigger than a chunk and cannot be smaller than a block
blocksize (int) – int [2**22] Number of bytes for a block. Within each chunk, we write a smaller block for each API call. This block cannot be bigger than a chunk
extra_upload_options (dict[str, Any] | None) – Extra upload options to add to the hook upload method
azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection