• Home
• Python API Reference
• airflow.hooks
• airflow.hooks.S3_hook

airflow.hooks.S3_hook

Module Contents

class airflow.hooks.S3_hook.S3Hook

Bases: airflow.contrib.hooks.aws_hook.AwsHook

Interact with AWS S3, using the boto3 library.

get_conn(self)

static parse_s3_url(s3url)

check_for_bucket(self, bucket_name)

Check if bucket_name exists.

Parameters

bucket_name (str) – the name of the bucket

get_bucket(self, bucket_name)

Returns a boto3.S3.Bucket object

Parameters

bucket_name (str) – the name of the bucket

create_bucket(self, bucket_name, region_name=None)

Creates an Amazon S3 bucket.

Parameters

• bucket_name (str) – The name of the bucket

• region_name (str) – The name of the aws region in which to create the bucket.

check_for_prefix(self, bucket_name, prefix, delimiter)

Checks that a prefix exists in a bucket

Parameters

• bucket_name (str) – the name of the bucket

• prefix (str) – a key prefix

• delimiter (str) – the delimiter marks key hierarchy.

list_prefixes(self, bucket_name, prefix='', delimiter='', page_size=None, max_items=None)

Lists prefixes in a bucket under prefix

Parameters

• bucket_name (str) – the name of the bucket

• prefix (str) – a key prefix

• delimiter (str) – the delimiter marks key hierarchy.

• page_size (int) – pagination size

• max_items (int) – maximum items to return

list_keys(self, bucket_name, prefix='', delimiter='', page_size=None, max_items=None)

Lists keys in a bucket under prefix and not containing delimiter

Parameters

• bucket_name (str) – the name of the bucket

• prefix (str) – a key prefix

• delimiter (str) – the delimiter marks key hierarchy.

• page_size (int) – pagination size

• max_items (int) – maximum items to return

check_for_key(self, key, bucket_name=None)

Checks if a key exists in a bucket

Parameters

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which the file is stored

get_key(self, key, bucket_name=None)

Returns a boto3.s3.Object

Parameters

• key (str) – the path to the key

• bucket_name (str) – the name of the bucket

read_key(self, key, bucket_name=None)

Reads a key from S3

Parameters

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which the file is stored

select_key(self, key, bucket_name=None, expression='SELECT * FROM S3Object', expression_type='SQL', input_serialization=None, output_serialization=None)

Reads a key with S3 Select.

Parameters

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which the file is stored

• expression (str) – S3 Select expression

• expression_type (str) – S3 Select expression type

• input_serialization (dict) – S3 Select input data serialization format

• output_serialization (dict) – S3 Select output data serialization format

Returns

retrieved subset of original data by S3 Select

Return type

str

See also

For more details about S3 Select parameters: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content

check_for_wildcard_key(self, wildcard_key, bucket_name=None, delimiter='')

Checks that a key matching a wildcard expression exists in a bucket

Parameters

• wildcard_key (str) – the path to the key

• bucket_name (str) – the name of the bucket

• delimiter (str) – the delimiter marks key hierarchy

get_wildcard_key(self, wildcard_key, bucket_name=None, delimiter='')

Returns a boto3.s3.Object object matching the wildcard expression

Parameters

• wildcard_key (str) – the path to the key

• bucket_name (str) – the name of the bucket

• delimiter (str) – the delimiter marks key hierarchy

load_file(self, filename, key, bucket_name=None, replace=False, encrypt=False)

Loads a local file to S3

Parameters

• filename (str) – name of the file to load.

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which to store the file

• replace (bool) – A flag to decide whether or not to overwrite the key if it already exists. If replace is False and the key exists, an error will be raised.

• encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_string(self, string_data, key, bucket_name=None, replace=False, encrypt=False, encoding='utf-8')

Loads a string to S3

This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.

Parameters

• string_data (str) – str to set as content for the key.

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which to store the file

• replace (bool) – A flag to decide whether or not to overwrite the key if it already exists

• encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_bytes(self, bytes_data, key, bucket_name=None, replace=False, encrypt=False)

Loads bytes to S3

This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.

Parameters

• bytes_data (bytes) – bytes to set as content for the key.

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which to store the file

• replace (bool) – A flag to decide whether or not to overwrite the key if it already exists

• encrypt (bool) – If True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

load_file_obj(self, file_obj, key, bucket_name=None, replace=False, encrypt=False)

Loads a file object to S3

Parameters

• file_obj (file-like object) – The file-like object to set as the content for the S3 key.

• key (str) – S3 key that will point to the file

• bucket_name (str) – Name of the bucket in which to store the file

• replace (bool) – A flag that indicates whether to overwrite the key if it already exists.

• encrypt (bool) – If True, S3 encrypts the file on the server, and the file is stored in encrypted form at rest in S3.

copy_object(self, source_bucket_key, dest_bucket_key, source_bucket_name=None, dest_bucket_name=None, source_version_id=None)

Creates a copy of an object that is already stored in S3.

Note: the S3 connection used here needs to have access to both source and destination bucket/key.

Parameters

• source_bucket_key (str) –

  The key of the source object.

  It can be either full s3:// style url or relative path from root level.

  When it’s specified as a full s3:// url, please omit source_bucket_name.

• dest_bucket_key (str) –

  The key of the object to copy to.

  The convention to specify dest_bucket_key is the same as source_bucket_key.

• source_bucket_name (str) –

  Name of the S3 bucket where the source object is in.

  It should be omitted when source_bucket_key is provided as a full s3:// url.

• dest_bucket_name (str) –

  Name of the S3 bucket to where the object is copied.

  It should be omitted when dest_bucket_key is provided as a full s3:// url.

• source_version_id (str) – Version ID of the source object (OPTIONAL)

delete_objects(self, bucket, keys)

Parameters

• bucket (str) – Name of the bucket in which you are going to delete object(s)

• keys (str or list) –

  The key(s) to delete from S3 bucket.

  When keys is a string, it’s supposed to be the key name of the single object to delete.

  When keys is a list, it’s supposed to be the list of the keys to delete.

Previous Next

Was this entry helpful?