Amazon S3 Operators¶

Airflow to Amazon Simple Storage Service (S3) integration provides several operators to create and interact with S3 buckets.

S3KeySensor

S3KeySizeSensor

S3KeysUnchangedSensor

S3PrefixSensor

S3CreateBucketOperator

S3DeleteBucketOperator

S3DeleteBucketTaggingOperator

S3GetBucketTaggingOperator

S3PutBucketTaggingOperator

S3CopyObjectOperator

S3DeleteObjectsOperator

S3FileTransformOperator

S3ListOperator

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Setup Connection.

Create an Amazon S3 Bucket¶

To create an Amazon S3 bucket you can use S3CreateBucketOperator.

airflow/providers/amazon/aws/example_dags/example_s3_bucket.py[source]

create_bucket = S3CreateBucketOperator(
    task_id='s3_create_bucket',
    bucket_name=BUCKET_NAME,
)

Delete an Amazon S3 Bucket¶

To delete an Amazon S3 bucket you can use S3DeleteBucketOperator.

airflow/providers/amazon/aws/example_dags/example_s3_bucket.py[source]

delete_bucket = S3DeleteBucketOperator(
    task_id='s3_delete_bucket', bucket_name=BUCKET_NAME, force_delete=True
)

Set the tags for an Amazon S3 Bucket¶

To set the tags for an Amazon S3 bucket you can use S3PutBucketTaggingOperator.

airflow/providers/amazon/aws/example_dags/example_s3_bucket.py[source]

put_tagging = S3PutBucketTaggingOperator(
    task_id='s3_put_bucket_tagging',
    bucket_name=BUCKET_NAME,
    key=TAG_KEY,
    value=TAG_VALUE,
)

Get the tag of an Amazon S3 Bucket¶

To get the tag set associated with an Amazon S3 bucket you can use S3GetBucketTaggingOperator.

airflow/providers/amazon/aws/example_dags/example_s3_bucket.py[source]

get_tagging = S3GetBucketTaggingOperator(
    task_id='s3_get_bucket_tagging',
    bucket_name=BUCKET_NAME,
)

Delete the tags of an Amazon S3 Bucket¶

To delete the tags of an Amazon S3 bucket you can use S3DeleteBucketTaggingOperator.

airflow/providers/amazon/aws/example_dags/example_s3_bucket.py[source]

delete_tagging = S3DeleteBucketTaggingOperator(
    task_id='s3_delete_bucket_tagging',
    bucket_name=BUCKET_NAME,
)

Reference¶

For further information, look at:

Boto3 Library Documentation for S3