AWS: Amazon Web Services¶

Airflow has support for Amazon Web Services.

All hooks are based on airflow.providers.amazon.aws.hooks.base_aws.

Service operators and hooks¶

These integrations allow you to perform various operations within the Amazon Web Services.

Transfer operators and hooks¶

These integrations allow you to copy data from/to Amazon Web Services.

Amazon DynamoDB to Amazon Simple Storage Service (S3)¶

Source product documentation: Amazon DynamoDB
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.dynamodb_to_s3
Provider: apache-airflow-providers-amazon

Google Cloud Storage (GCS) to Amazon Simple Storage Service (S3)¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.gcs_to_s3
Provider: apache-airflow-providers-amazon

Amazon Glacier to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Glacier
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Amazon Glacier Transfer Operator
Python API: airflow.providers.amazon.aws.transfers.glacier_to_gcs
Provider: apache-airflow-providers-amazon

Google to Amazon Simple Storage Service (S3)¶

Source product documentation: Google
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Google API To S3 Transfer
Python API: airflow.providers.amazon.aws.transfers.google_api_to_s3
Provider: apache-airflow-providers-amazon

Apache Hive to Amazon DynamoDB¶

Source product documentation: Apache Hive
Target product documentation: Amazon DynamoDB
Python API: airflow.providers.amazon.aws.transfers.hive_to_dynamodb
Provider: apache-airflow-providers-amazon

Internet Message Access Protocol (IMAP) to Amazon Simple Storage Service (S3)¶

Source product documentation: Internet Message Access Protocol (IMAP)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Imap Attachment To S3 Operator
Python API: airflow.providers.amazon.aws.transfers.imap_attachment_to_s3
Provider: apache-airflow-providers-amazon

MongoDB to Amazon Simple Storage Service (S3)¶

Source product documentation: MongoDB
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.mongo_to_s3
Provider: apache-airflow-providers-amazon

MySQL to Amazon Simple Storage Service (S3)¶

Source product documentation: MySQL
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.mysql_to_s3
Provider: apache-airflow-providers-amazon

Amazon Redshift to Amazon Simple Storage Service (S3)¶

Source product documentation: Amazon Redshift
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.redshift_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Amazon Redshift¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Amazon Redshift
Operator guide: S3 To Redshift Transfer Operator
Python API: airflow.providers.amazon.aws.transfers.s3_to_redshift
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to SSH File Transfer Protocol (SFTP)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: SSH File Transfer Protocol (SFTP)
Python API: airflow.providers.amazon.aws.transfers.s3_to_sftp
Provider: apache-airflow-providers-amazon

SSH File Transfer Protocol (SFTP) to Amazon Simple Storage Service (S3)¶

Source product documentation: SSH File Transfer Protocol (SFTP)
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.sftp_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to File Transfer Protocol (FTP)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: File Transfer Protocol (FTP)
Python API: airflow.providers.amazon.aws.transfers.s3_to_ftp
Provider: apache-airflow-providers-amazon

Exasol to Amazon Simple Storage Service (S3)¶

Source product documentation: Exasol
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.exasol_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Apache Hive¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Apache Hive
Python API: airflow.providers.apache.hive.transfers.s3_to_hive
Provider: apache-airflow-providers-apache-hive

Amazon Simple Storage Service (S3) to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Transfer Data from Amazon S3 to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.s3_to_gcs
Provider: apache-airflow-providers-google

Amazon Simple Storage Service (S3) to MySQL¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: MySQL
Python API: airflow.providers.mysql.transfers.s3_to_mysql
Provider: apache-airflow-providers-mysql

Amazon Simple Storage Service (S3) to Snowflake¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Snowflake
Operator guide: S3ToSnowflakeOperator
Python API: airflow.providers.snowflake.transfers.s3_to_snowflake
Provider: apache-airflow-providers-snowflake

AWS: Amazon Web Services¶

Service operators and hooks¶

Amazon Athena¶

Amazon CloudFormation¶

Amazon CloudWatch Logs¶

Amazon DataSync¶

Amazon DynamoDB¶

Amazon EC2¶

Amazon ECS¶

Amazon ElastiCache¶

Amazon EMR¶

Amazon Glacier¶

Amazon Kinesis Data Firehose¶

Amazon Redshift¶

Amazon SageMaker¶

Amazon SecretsManager¶

Amazon Simple Email Service (SES)¶

Amazon Simple Notification Service (SNS)¶

Amazon Simple Queue Service (SQS)¶

Amazon Simple Storage Service (S3)¶

Amazon Web Services¶

AWS Batch¶

AWS Glue¶

AWS Lambda¶

AWS Step Functions¶