AWS: Amazon Web Services¶

Airflow has support for Amazon Web Services.

All hooks are based on airflow.providers.amazon.aws.hooks.base_aws.

Services¶

These integrations allow you to perform various operations within the Amazon Web Services.

Transfers¶

These integrations allow you to copy data from/to Amazon Web Services.

Amazon DynamoDB to Amazon Simple Storage Service (S3)¶

Source product documentation: Amazon DynamoDB
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.dynamodb_to_s3
Provider: apache-airflow-providers-amazon

Google Cloud Storage (GCS) to Amazon Simple Storage Service (S3)¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.gcs_to_s3
Provider: apache-airflow-providers-amazon

Amazon Glacier to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Glacier
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Amazon Glacier Transfer Operator
Python API: airflow.providers.amazon.aws.transfers.glacier_to_gcs
Provider: apache-airflow-providers-amazon

Google to Amazon Simple Storage Service (S3)¶

Source product documentation: Google
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Google API To S3 Transfer
Python API: airflow.providers.amazon.aws.transfers.google_api_to_s3
Provider: apache-airflow-providers-amazon

Apache Hive to Amazon DynamoDB¶

Source product documentation: Apache Hive
Target product documentation: Amazon DynamoDB
Python API: airflow.providers.amazon.aws.transfers.hive_to_dynamodb
Provider: apache-airflow-providers-amazon

Internet Message Access Protocol (IMAP) to Amazon Simple Storage Service (S3)¶

Source product documentation: Internet Message Access Protocol (IMAP)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Imap Attachment To S3 Operator
Python API: airflow.providers.amazon.aws.transfers.imap_attachment_to_s3
Provider: apache-airflow-providers-amazon

MongoDB to Amazon Simple Storage Service (S3)¶

Source product documentation: MongoDB
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.mongo_to_s3
Provider: apache-airflow-providers-amazon

MySQL to Amazon Simple Storage Service (S3)¶

Source product documentation: MySQL
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.mysql_to_s3
Provider: apache-airflow-providers-amazon

Amazon Redshift to Amazon Simple Storage Service (S3)¶

Source product documentation: Amazon Redshift
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.redshift_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Amazon Redshift¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Amazon Redshift
Operator guide: S3 To Redshift Transfer Operator
Python API: airflow.providers.amazon.aws.transfers.s3_to_redshift
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to SSH File Transfer Protocol (SFTP)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: SSH File Transfer Protocol (SFTP)
Operator guide: Amazon S3 to SFTP Transfer Operator
Python API: airflow.providers.amazon.aws.transfers.s3_to_sftp
Provider: apache-airflow-providers-amazon

SSH File Transfer Protocol (SFTP) to Amazon Simple Storage Service (S3)¶

Source product documentation: SSH File Transfer Protocol (SFTP)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Amazon SFTP to S3 Transfer Operator
Python API: airflow.providers.amazon.aws.transfers.sftp_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to File Transfer Protocol (FTP)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: File Transfer Protocol (FTP)
Python API: airflow.providers.amazon.aws.transfers.s3_to_ftp
Provider: apache-airflow-providers-amazon

Exasol to Amazon Simple Storage Service (S3)¶

Source product documentation: Exasol
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.exasol_to_s3
Provider: apache-airflow-providers-amazon

File Transfer Protocol (FTP) to Amazon Simple Storage Service (S3)¶

Source product documentation: File Transfer Protocol (FTP)
Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.ftp_to_s3
Provider: apache-airflow-providers-amazon

Salesforce to Amazon Simple Storage Service (S3)¶

Source product documentation: Salesforce
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Salesforce To S3 Operator
Python API: airflow.providers.amazon.aws.transfers.salesforce_to_s3
Provider: apache-airflow-providers-amazon

Local to Amazon Simple Storage Service (S3)¶

Target product documentation: Amazon Simple Storage Service (S3)
Python API: airflow.providers.amazon.aws.transfers.local_to_s3
Provider: apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Apache Hive¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Apache Hive
Python API: airflow.providers.apache.hive.transfers.s3_to_hive
Provider: apache-airflow-providers-apache-hive

Amazon Simple Storage Service (S3) to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Transfer Data from Amazon S3 to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.s3_to_gcs
Provider: apache-airflow-providers-google

Amazon Simple Storage Service (S3) to MySQL¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: MySQL
Python API: airflow.providers.mysql.transfers.s3_to_mysql
Provider: apache-airflow-providers-mysql

Amazon Simple Storage Service (S3) to Snowflake¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Snowflake
Operator guide: S3ToSnowflakeOperator
Python API: airflow.providers.snowflake.transfers.s3_to_snowflake
Provider: apache-airflow-providers-snowflake

AWS: Amazon Web Services¶

Services¶

Amazon Athena¶

Amazon CloudFormation¶

Amazon CloudWatch Logs¶

Amazon DataSync¶

Amazon DynamoDB¶

Amazon EC2¶

Amazon ECS¶

Amazon Elastic Kubernetes Service (EKS)¶

Amazon ElastiCache¶

Amazon EMR¶

Amazon EMR on EKS¶

Amazon Glacier¶

Amazon Kinesis Data Firehose¶

Amazon Redshift¶

Amazon SageMaker¶

Amazon SecretsManager¶

Amazon Simple Email Service (SES)¶

Amazon Simple Notification Service (SNS)¶

Amazon Simple Queue Service (SQS)¶

Amazon Simple Storage Service (S3)¶

Amazon Web Services¶

AWS Batch¶

AWS Database Migration Service¶

AWS Glue¶

AWS Lambda¶

AWS Step Functions¶