Azure DataLake Storage Operators¶

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AZURE PORTAL or AZURE CLI.
Install API libraries via pip.
pip install 'apache-airflow[azure]'
Detailed information is available Installation of Airflow®
Setup Connection.

ADLSCreateObjectOperator¶

ADLSCreateObjectOperator allows you to upload data to Azure DataLake Storage

Below is an example of using this operator to upload data to ADL.

tests/system/microsoft/azure/example_adls_create.py[source]

    upload_data = ADLSCreateObjectOperator(
        task_id="upload_data",
        file_system_name="Fabric",
        file_name=REMOTE_FILE_PATH,
        data="Hello world",
        replace=True,
    )

ADLSDeleteOperator¶

Use the ADLSDeleteOperator to remove file(s) from Azure DataLake Storage

Below is an example of using this operator to delete a file from ADL.

tests/system/microsoft/azure/example_adls_delete.py[source]

    remove_file = ADLSDeleteOperator(task_id="delete_task", path=REMOTE_FILE_PATH, recursive=True)

ADLSListOperator¶

Use the ADLSListOperator to list all file(s) from Azure DataLake Storage

Below is an example of using this operator to list files from ADL.

tests/system/microsoft/azure/example_adls_list.py[source]

    adls_files = ADLSListOperator(
        task_id="adls_files",
        path="folder/output/*.parquet",
        azure_data_lake_conn_id="azure_data_lake_default",
    )

Reference¶

For further information, look at:

Azure Data lake Storage Documentation