airflow.contrib.hooks.aws_glue_catalog_hook

This module contains AWS Glue Catalog Hook

Module Contents

class airflow.contrib.hooks.aws_glue_catalog_hook.AwsGlueCatalogHook(aws_conn_id='aws_default', region_name=None, *args, **kwargs)[source]

Bases: airflow.contrib.hooks.aws_hook.AwsHook

Interact with AWS Glue Catalog

Parameters
  • aws_conn_id (str) – ID of the Airflow connection where credentials and extra configuration are stored

  • region_name (str) – aws region name (example: us-east-1)

get_conn(self)[source]

Returns glue connection object.

get_partitions(self, database_name, table_name, expression='', page_size=None, max_items=None)[source]

Retrieves the partition values for a table.

Parameters
Returns

set of partition values where each value is a tuple since a partition may be composed of multiple columns. For example: {('2018-01-01','1'), ('2018-01-01','2')}

check_for_partition(self, database_name, table_name, expression)[source]

Checks whether a partition exists

Parameters
  • database_name (str) – Name of hive database (schema) @table belongs to

  • table_name (str) – Name of hive table @partition belongs to

Expression

Expression that matches the partitions to check for (eg a = ‘b’ AND c = ‘d’)

Return type

bool

>>> hook = AwsGlueCatalogHook()
>>> t = 'static_babynames_partitioned'
>>> hook.check_for_partition('airflow', t, "ds='2015-01-01'")
True
get_table(self, database_name, table_name)[source]

Get the information of the table

Parameters
  • database_name (str) – Name of hive database (schema) @table belongs to

  • table_name (str) – Name of hive table

Return type

dict

>>> hook = AwsGlueCatalogHook()
>>> r = hook.get_table('db', 'table_foo')
>>> r['Name'] = 'table_foo'
get_table_location(self, database_name, table_name)[source]

Get the physical location of the table

Parameters
  • database_name (str) – Name of hive database (schema) @table belongs to

  • table_name (str) – Name of hive table

Returns

str

Was this entry helpful?