airflow.providers.microsoft.azure.operators.adls

Module Contents

Classes

ADLSDeleteOperator

Delete files in the specified path.

ADLSListOperator

List all files from the specified path.
class airflow.providers.microsoft.azure.operators.adls.ADLSDeleteOperator(*, path, recursive=False, ignore_not_found=True, azure_data_lake_conn_id='azure_data_lake_default', **kwargs)[source]

Bases: airflow.models.BaseOperator

Delete files in the specified path.

See also

For more information on how to use this operator, take a look at the guide: ADLSDeleteOperator

Parameters

  • path (str) – A directory or file to remove

  • recursive (bool) – Whether to loop into directories in the location and remove the files

  • ignore_not_found (bool) – Whether to raise error if file to delete is not found

  • azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection.

template_fields: Sequence[str] = ('path',)[source]
ui_color = '#901dd2'[source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.microsoft.azure.operators.adls.ADLSListOperator(*, path, azure_data_lake_conn_id='azure_data_lake_default', **kwargs)[source]

Bases: airflow.models.BaseOperator

List all files from the specified path.

This operator returns a python list with the names of files which can be used by

xcom in the downstream tasks.

Parameters

  • path (str) – The Azure Data Lake path to find the objects. Supports glob strings (templated)

  • azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection.

Example:

The following Operator would list all the Parquet files from folder/output/ folder in the specified ADLS account

adls_files = ADLSListOperator(
    task_id='adls_files',
    path='folder/output/*.parquet',
    azure_data_lake_conn_id='azure_data_lake_default'
)
template_fields: Sequence[str] = ('path',)[source]
ui_color = '#901dd2'[source]
execute(context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?