airflow.providers.amazon.aws.operators.s3_list_prefixes

Module Contents

class airflow.providers.amazon.aws.operators.s3_list_prefixes.S3ListPrefixesOperator(*, bucket: str, prefix: str, delimiter: str, aws_conn_id: str = 'aws_default', verify: Optional[Union[str, bool]] = None, **kwargs)[source]

Bases: airflow.models.BaseOperator

List all subfolders from the bucket with the given string prefix in name.

This operator returns a python list with the name of all subfolders which can be used by xcom in the downstream task.

Parameters

  • bucket (str) -- The S3 bucket where to find the subfolders. (templated)

  • prefix (str) -- Prefix string to filter the subfolders whose name begin with such prefix. (templated)

  • delimiter (str) -- the delimiter marks subfolder hierarchy. (templated)

  • aws_conn_id (str) -- The connection ID to use when connecting to S3 storage.

  • verify (bool or str) --

    Whether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values:

    • False: do not validate SSL certificates. SSL will still be used

      (unless use_ssl is False), but SSL certificates will not be verified.

    • path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.

      You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.

Example:

The following operator would list all the subfolders from the S3 customers/2018/04/ prefix in the data bucket.

s3_file = S3ListPrefixesOperator(
    task_id='list_s3_prefixes',
    bucket='data',
    prefix='customers/2018/04/',
    delimiter='/',
    aws_conn_id='aws_customers_conn'
)
template_fields :Iterable[str] = ['bucket', 'prefix', 'delimiter'][source]
ui_color = #ffd700[source]
execute(self, context)[source]

Was this entry helpful?