airflow.providers.amazon.aws.operators.glue_crawler

Module Contents

Classes

GlueCrawlerOperator

Creates, updates and triggers an AWS Glue Crawler.
class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerOperator(config, aws_conn_id='aws_default', region_name=None, poll_interval=5, wait_for_completion=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]

Bases: airflow.models.BaseOperator

Creates, updates and triggers an AWS Glue Crawler.

AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.

See also

For more information on how to use this operator, take a look at the guide: Create an AWS Glue crawler

Parameters

  • config – Configurations for the AWS Glue crawler

  • aws_conn_id – aws connection to use

  • poll_interval (int) – Time (in seconds) to wait between two consecutive calls to check crawler status

  • wait_for_completion (bool) – Whether to wait for crawl execution completion. (default: True)

  • deferrable (bool) – If True, the operator will wait asynchronously for the crawl to complete. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)

template_fields: Sequence[str] = ('config',)[source]
ui_color = '#ededed'[source]
hook()[source]

Create and return a GlueCrawlerHook.

execute(context)[source]

Executes AWS Glue Crawler from Airflow.

Returns

the name of the current glue crawler.

execute_complete(context, event=None)[source]

Was this entry helpful?