airflow.contrib.operators.adls_list_operator

Module Contents

class airflow.contrib.operators.adls_list_operator.AzureDataLakeStorageListOperator(path, azure_data_lake_conn_id='azure_data_lake_default', *args, **kwargs)[source]

Bases: airflow.models.BaseOperator

List all files from the specified path

This operator returns a python list with the names of files which can be used by

xcom in the downstream tasks.

Parameters

  • path (str) – The Azure Data Lake path to find the objects. Supports glob strings (templated)

  • azure_data_lake_conn_id (str) – The connection ID to use when connecting to Azure Data Lake Storage.

Example:

The following Operator would list all the Parquet files from folder/output/ folder in the specified ADLS account

adls_files = AzureDataLakeStorageListOperator(
    task_id='adls_files',
    path='folder/output/*.parquet',
    azure_data_lake_conn_id='azure_data_lake_default'
)
template_fields :Iterable[str] = ['path'][source]
ui_color = #901dd2[source]
execute(self, context)[source]

Was this entry helpful?