airflow.hooks.S3_hook

Module Contents

class airflow.hooks.S3_hook.S3Hook[source]

Bases: airflow.contrib.hooks.aws_hook.AwsHook

Interact with AWS S3, using the boto3 library.

get_conn(self)[source]
static parse_s3_url(s3url)[source]
check_for_bucket(self, bucket_name)[source]

Check if bucket_name exists.

Parameters

bucket_name (str) – the name of the bucket

get_bucket(self, bucket_name)[source]

Returns a boto3.S3.Bucket object

Parameters

bucket_name (str) – the name of the bucket

create_bucket(self, bucket_name, region_name=None)[source]

Creates an Amazon S3 bucket.

Parameters

  • bucket_name (str) – The name of the bucket

  • region_name (str) – The name of the aws region in which to create the bucket.

check_for_prefix(self, bucket_name, prefix, delimiter)[source]

Checks that a prefix exists in a bucket

Parameters

  • bucket_name (str) – the name of the bucket

  • prefix (str) – a key prefix

  • delimiter (str) – the delimiter marks key hierarchy.

list_prefixes(self, bucket_name, prefix='', delimiter='', page_size=None, max_items=None)[source]

Lists prefixes in a bucket under prefix

Parameters

  • bucket_name (str) – the name of the bucket

  • prefix (str) – a key prefix

  • delimiter (str) – the delimiter marks key hierarchy.

  • page_size (int) – pagination size

  • max_items (int) – maximum items to return

list_keys(self, bucket_name, prefix='', delimiter='', page_size=None, max_items=None)[source]

Lists keys in a bucket under prefix and not containing delimiter

Parameters

  • bucket_name (str) – the name of the bucket

  • prefix (str) – a key prefix

  • delimiter (str) – the delimiter marks key hierarchy.

  • page_size (int) – pagination size

  • max_items (int) – maximum items to return

check_for_key(self, key, bucket_name=None)[source]

Checks if a key exists in a bucket

Parameters

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which the file is stored

get_key(self, key, bucket_name=None)[source]

Returns a boto3.s3.Object

Parameters

  • key (str) – the path to the key

  • bucket_name (str) – the name of the bucket

read_key(self, key, bucket_name=None)[source]

Reads a key from S3

Parameters

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which the file is stored

select_key(self, key, bucket_name=None, expression='SELECT * FROM S3Object', expression_type='SQL', input_serialization=None, output_serialization=None)[source]

Reads a key with S3 Select.

Parameters

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which the file is stored

  • expression (str) – S3 Select expression

  • expression_type (str) – S3 Select expression type

  • input_serialization (dict) – S3 Select input data serialization format

  • output_serialization (dict) – S3 Select output data serialization format

Returns

retrieved subset of original data by S3 Select

Return type

str

See also

For more details about S3 Select parameters: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content

check_for_wildcard_key(self, wildcard_key, bucket_name=None, delimiter='')[source]

Checks that a key matching a wildcard expression exists in a bucket

Parameters

  • wildcard_key (str) – the path to the key

  • bucket_name (str) – the name of the bucket

  • delimiter (str) – the delimiter marks key hierarchy

get_wildcard_key(self, wildcard_key, bucket_name=None, delimiter='')[source]

Returns a boto3.s3.Object object matching the wildcard expression

Parameters

  • wildcard_key (str) – the path to the key

  • bucket_name (str) – the name of the bucket

  • delimiter (str) – the delimiter marks key hierarchy

load_file(self, filename, key, bucket_name=None, replace=False, encrypt=False)[source]

Loads a local file to S3

Parameters

  • filename (str) – name of the file to load.

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which to store the file

  • replace (bool) – A flag to decide whether or not to overwrite the key if it already exists. If replace is False and the key exists, an error will be raised.

  • encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_string(self, string_data, key, bucket_name=None, replace=False, encrypt=False, encoding='utf-8')[source]

Loads a string to S3

This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.

Parameters

  • string_data (str) – str to set as content for the key.

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which to store the file

  • replace (bool) – A flag to decide whether or not to overwrite the key if it already exists

  • encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_bytes(self, bytes_data, key, bucket_name=None, replace=False, encrypt=False)[source]

Loads bytes to S3

This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.

Parameters

  • bytes_data (bytes) – bytes to set as content for the key.

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which to store the file

  • replace (bool) – A flag to decide whether or not to overwrite the key if it already exists

  • encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_file_obj(self, file_obj, key, bucket_name=None, replace=False, encrypt=False)[source]

Loads a file object to S3

Parameters

  • file_obj (file-like object) – The file-like object to set as the content for the S3 key.

  • key (str) – S3 key that will point to the file

  • bucket_name (str) – Name of the bucket in which to store the file

  • replace (bool) – A flag that indicates whether to overwrite the key if it already exists.

  • encrypt (bool) – If True, S3 encrypts the file on the server, and the file is stored in encrypted form at rest in S3.

copy_object(self, source_bucket_key, dest_bucket_key, source_bucket_name=None, dest_bucket_name=None, source_version_id=None)[source]

Creates a copy of an object that is already stored in S3.

Note: the S3 connection used here needs to have access to both source and destination bucket/key.

Parameters

  • source_bucket_key (str) –

    The key of the source object.

    It can be either full s3:// style url or relative path from root level.

    When it’s specified as a full s3:// url, please omit source_bucket_name.

  • dest_bucket_key (str) –

    The key of the object to copy to.

    The convention to specify dest_bucket_key is the same as source_bucket_key.

  • source_bucket_name (str) –

    Name of the S3 bucket where the source object is in.

    It should be omitted when source_bucket_key is provided as a full s3:// url.

  • dest_bucket_name (str) –

    Name of the S3 bucket to where the object is copied.

    It should be omitted when dest_bucket_key is provided as a full s3:// url.

  • source_version_id (str) – Version ID of the source object (OPTIONAL)

delete_objects(self, bucket, keys)[source]
Parameters

  • bucket (str) – Name of the bucket in which you are going to delete object(s)

  • keys (str or list) –

    The key(s) to delete from S3 bucket.

    When keys is a string, it’s supposed to be the key name of the single object to delete.

    When keys is a list, it’s supposed to be the list of the keys to delete.