airflow.providers.amazon.aws.operators.glue_crawler

Module Contents

Classes

GlueCrawlerOperator

Creates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless
class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerOperator(config, aws_conn_id='aws_default', region_name=None, poll_interval=5, wait_for_completion=True, **kwargs)[source]

Bases: airflow.models.BaseOperator

Creates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.

See also

For more information on how to use this operator, take a look at the guide: Create an AWS Glue crawler

Parameters

  • config – Configurations for the AWS Glue crawler

  • aws_conn_id – aws connection to use

  • poll_interval (int) – Time (in seconds) to wait between two consecutive calls to check crawler status

  • wait_for_completion (bool) – Whether or not wait for crawl execution completion. (default: True)

template_fields: Sequence[str] = ('config',)[source]
ui_color = '#ededed'[source]
hook()[source]

Create and return an GlueCrawlerHook.

execute(context)[source]

Executes AWS Glue Crawler from Airflow

Returns

the name of the current glue crawler.

Was this entry helpful?