Amazon S3 Operators¶
Prerequisite Tasks¶
To use these operators, you must do a few things:
Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'
Detailed information is available Installation
Overview¶
Airflow to Amazon Simple Storage Service (S3) integration provides several operators to create and interact with S3 buckets.
Two example_dags are provided which showcase these operators in action.
example_s3_bucket.py
example_s3_bucket_tagging.py
Create and Delete Amazon S3 Buckets¶
Purpose¶
This example dag example_s3_bucket.py uses S3CreateBucketOperator and S3DeleteBucketOperator to create a
new S3 bucket with a given bucket name then delete it.
Defining tasks¶
In the following code we create a new bucket, add keys, and then delete the bucket.
create_bucket = S3CreateBucketOperator(
task_id='s3_bucket_dag_create',
bucket_name=BUCKET_NAME,
region_name='us-east-1',
)
add_keys_to_bucket = PythonOperator(
task_id="s3_bucket_dag_add_keys_to_bucket", python_callable=upload_keys
)
delete_bucket = S3DeleteBucketOperator(
task_id='s3_bucket_dag_delete',
bucket_name=BUCKET_NAME,
force_delete=True,
)
Using Amazon S3 Bucket Tagging¶
Purpose¶
This example dag example_s3_bucket_tagging.py uses S3DeleteBucketTaggingOperator, S3GetBucketTaggingOperator,
and S3PutBucketTaggingOperator to create a new S3 bucket, apply tagging, get tagging, delete tagging, then delete the bucket.
Defining tasks¶
In the following code we create a new S3 bucket, apply tagging, get tagging, delete tagging, then delete the bucket.
get_tagging = S3GetBucketTaggingOperator(
task_id='s3_bucket_tagging_dag_get_tagging', bucket_name=BUCKET_NAME
)
put_tagging = S3PutBucketTaggingOperator(
task_id='s3_bucket_tagging_dag_put_tagging', bucket_name=BUCKET_NAME, key=TAG_KEY, value=TAG_VALUE
)
delete_tagging = S3DeleteBucketTaggingOperator(
task_id='s3_bucket_tagging_dag_delete_tagging', bucket_name=BUCKET_NAME
)