airflow.providers.amazon.aws.sensors.glue_catalog_partition
¶
Module Contents¶
-
class
airflow.providers.amazon.aws.sensors.glue_catalog_partition.
AwsGlueCatalogPartitionSensor
(*, table_name: str, expression: str = "ds='{{ ds }}'", aws_conn_id: str = 'aws_default', region_name: Optional[str] = None, database_name: str = 'default', poke_interval: int = 60 * 3, **kwargs)[source]¶ Bases:
airflow.sensors.base.BaseSensorOperator
Waits for a partition to show up in AWS Glue Catalog.
- Parameters
table_name (str) -- The name of the table to wait for, supports the dot notation (my_database.my_table)
expression (str) -- The partition clause to wait for. This is passed as is to the AWS Glue Catalog API's get_partitions function, and supports SQL like notation as in
ds='2015-01-01' AND type='value'
and comparison operators as in"ds>=2015-01-01"
. See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html #aws-glue-api-catalog-partitions-GetPartitionsaws_conn_id (str) -- ID of the Airflow connection where credentials and extra configuration are stored
region_name (str) -- Optional aws region name (example: us-east-1). Uses region from connection if not specified.
database_name (str) -- The name of the catalog database where the partitions reside.
poke_interval (int) -- Time in seconds that the job should wait in between each tries