airflow.providers.amazon.aws.operators.glue_crawler

Module Contents

class airflow.providers.amazon.aws.operators.glue_crawler.AwsGlueCrawlerOperator(config, aws_conn_id='aws_default', poll_interval: int = 5, **kwargs)[source]

Bases: airflow.models.BaseOperator

Creates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.

Parameters

  • config (dict) -- Configurations for the AWS Glue crawler

  • aws_conn_id (Optional[str]) -- aws connection to use

  • poll_interval (Optional[int]) -- Time (in seconds) to wait between two consecutive calls to check crawler status

ui_color = #ededed[source]
hook(self)[source]

Create and return an AwsGlueCrawlerHook.

execute(self, context)[source]

Executes AWS Glue Crawler from Airflow

Returns

the name of the current glue crawler.

Was this entry helpful?