airflow.providers.amazon.aws.hooks.glue_catalog
¶
This module contains AWS Glue Catalog Hook
Module Contents¶
Classes¶
Interact with AWS Glue Catalog |
|
This hook is deprecated. |
- class airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook(*args, **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook
Interact with AWS Glue Catalog
Additional arguments (such as
aws_conn_id
) may be specified and are passed down to the underlying AwsBaseHook.See also
- get_partitions(self, database_name, table_name, expression='', page_size=None, max_items=None)[source]¶
Retrieves the partition values for a table.
- Parameters
database_name (str) – The name of the catalog database where the partitions reside.
table_name (str) – The name of the partitions’ table.
expression (str) – An expression filtering the partitions to be returned. Please see official AWS documentation for further information. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-GetPartitions
page_size (Optional[int]) – pagination size
max_items (Optional[int]) – maximum items to return
- Returns
set of partition values where each value is a tuple since a partition may be composed of multiple columns. For example:
{('2018-01-01','1'), ('2018-01-01','2')}
- Return type
Set[tuple]
- check_for_partition(self, database_name, table_name, expression)[source]¶
Checks whether a partition exists
- Parameters
- Expression
Expression that matches the partitions to check for (eg a = ‘b’ AND c = ‘d’)
- Return type
>>> hook = GlueCatalogHook() >>> t = 'static_babynames_partitioned' >>> hook.check_for_partition('airflow', t, "ds='2015-01-01'") True
- class airflow.providers.amazon.aws.hooks.glue_catalog.AwsGlueCatalogHook(*args, **kwargs)[source]¶
Bases:
GlueCatalogHook
This hook is deprecated. Please use
airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook
.