AWS: Amazon Web Services¶

Airflow has support for Amazon Web Services.

All hooks are based on airflow.providers.amazon.aws.hooks.base_aws.

Services¶

These integrations allow you to perform various operations within the Amazon Web Services.

Transfers¶

These integrations allow you to copy data from/to Amazon Web Services.

Amazon DynamoDB to Amazon Simple Storage Service (S3)¶

Source product documentation: Amazon DynamoDB
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Amazon DynamoDB to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.dynamodb_to_s3
Provider: apache-airflow-providers-amazon

Google Cloud Storage (GCS) to Amazon Simple Storage Service (S3)¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Google Cloud Storage to Amazon S3 transfer operator
Python API: airflow.providers.amazon.aws.transfers.gcs_to_s3
Provider: apache-airflow-providers-amazon

Amazon Glacier to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Glacier
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Amazon S3 Glacier to GCS
Python API: airflow.providers.amazon.aws.transfers.glacier_to_gcs
Provider: apache-airflow-providers-amazon

Google to Amazon Simple Storage Service (S3)¶

Source product documentation: Google
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Google API to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.google_api_to_s3
Provider: apache-airflow-providers-amazon

Apache Hive to Amazon DynamoDB¶

Source product documentation: Apache Hive
Target product documentation: Amazon DynamoDB
Operator guide: Apache Hive to Amazon DynamoDB
Python API: airflow.providers.amazon.aws.transfers.hive_to_dynamodb
Provider: apache-airflow-providers-amazon

Internet Message Access Protocol (IMAP) to Amazon Simple Storage Service (S3)¶

Source product documentation: Internet Message Access Protocol (IMAP)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Imap Attachment to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.imap_attachment_to_s3
Provider: apache-airflow-providers-amazon

MongoDB to Amazon Simple Storage Service (S3)¶

Source product documentation: MongoDB
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: MongoDB to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.mongo_to_s3
Provider: apache-airflow-providers-amazon

MySQL to Amazon Simple Storage Service (S3)¶

Source product documentation: MySQL
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: SQL to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.mysql_to_s3
Provider: apache-airflow-providers-amazon

Amazon Redshift to Amazon Simple Storage Service (S3)¶

Source product documentation: Amazon Redshift
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Amazon Redshift to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.redshift_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Amazon Redshift¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Amazon Redshift
Operator guide: Amazon S3 to Amazon Redshift
Python API: airflow.providers.amazon.aws.transfers.s3_to_redshift
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to SSH File Transfer Protocol (SFTP)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: SSH File Transfer Protocol (SFTP)
Operator guide: Amazon S3 to SFTP
Python API: airflow.providers.amazon.aws.transfers.s3_to_sftp
Provider: apache-airflow-providers-amazon

SSH File Transfer Protocol (SFTP) to Amazon Simple Storage Service (S3)¶

Source product documentation: SSH File Transfer Protocol (SFTP)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: SFTP to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.sftp_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to File Transfer Protocol (FTP)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: File Transfer Protocol (FTP)
Operator guide: Amazon S3 to FTP
Python API: airflow.providers.amazon.aws.transfers.s3_to_ftp
Provider: apache-airflow-providers-amazon

Exasol to Amazon Simple Storage Service (S3)¶

Source product documentation: Exasol
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.exasol_to_s3
Provider: apache-airflow-providers-amazon

File Transfer Protocol (FTP) to Amazon Simple Storage Service (S3)¶

Source product documentation: File Transfer Protocol (FTP)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: FTP to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.ftp_to_s3
Provider: apache-airflow-providers-amazon

Salesforce to Amazon Simple Storage Service (S3)¶

Source product documentation: Salesforce
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Salesforce to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.salesforce_to_s3
Provider: apache-airflow-providers-amazon

Local to Amazon Simple Storage Service (S3)¶

Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Local Filesystem to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.local_to_s3
Provider: apache-airflow-providers-amazon

SQL to Amazon Simple Storage Service (S3)¶

Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: SQL to Amazon S3
Python API: airflow.providers.amazon.aws.transfers.sql_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Apache Hive¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Apache Hive
Python API: airflow.providers.apache.hive.transfers.s3_to_hive
Provider: apache-airflow-providers-apache-hive

Amazon Simple Storage Service (S3) to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Transfer Data from Amazon S3 to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.s3_to_gcs
Provider: apache-airflow-providers-google

Amazon Simple Storage Service (S3) to MySQL¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: MySQL
Python API: airflow.providers.mysql.transfers.s3_to_mysql
Provider: apache-airflow-providers-mysql

Amazon Simple Storage Service (S3) to Snowflake¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Snowflake
Operator guide: S3ToSnowflakeOperator
Python API: airflow.providers.snowflake.transfers.s3_to_snowflake
Provider: apache-airflow-providers-snowflake

AWS: Amazon Web Services¶

Services¶

Amazon Athena¶

Amazon CloudFormation¶

Amazon CloudWatch Logs¶

Amazon DataSync¶

Amazon DynamoDB¶

Amazon EC2¶

Amazon ECS¶

Amazon Elastic Kubernetes Service (EKS)¶

Amazon ElastiCache¶

Amazon EMR¶

Amazon EMR on EKS¶

Amazon Glacier¶

Amazon Kinesis Data Firehose¶

Amazon QuickSight¶

Amazon RDS¶

Amazon Redshift¶

Amazon SageMaker¶

Amazon SecretsManager¶

Amazon Simple Email Service (SES)¶

Amazon Simple Notification Service (SNS)¶

Amazon Simple Queue Service (SQS)¶

Amazon Simple Storage Service (S3)¶

Amazon Web Services¶

AWS Batch¶

AWS Database Migration Service¶

AWS Glue¶

AWS Lambda¶

AWS Security Token Service (STS)¶

AWS Step Functions¶